2023-07-14T18:35:43.728 INFO:root:teuthology version: 0.0.1.dev145+gaa6a80e 2023-07-14T18:35:43.729 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038 --description fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} --verbose --name pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi --owner scheduled_pdonnell@teuthology -- /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/orig.config.yaml 2023-07-14T18:35:43.784 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T18:35:48.896 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038 branch: wip-pdonnell-testing-20230714.134335 description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} email: pdonnell@redhat.com first_in_suite: false job_id: '7338038' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230714.134335 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec kclient: client.0: mount_subvol_num: 1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230714.134335 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 5179 sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec sleep_before_teardown: 0 subset: 285/512 suite: fs:workload suite_branch: wip-pdonnell-testing-20230714.134335 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi078.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi102.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools:rhel8 --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - ceph-fuse: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs _nogroup distributed 1 - check-counter: {} - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: aa6a80e6cc655fff48452a7eca02070e61d074fc timestamp: 2023-07-14_16:44:47 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-07-14T18:35:48.896 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa; will attempt to use it 2023-07-14T18:35:48.897 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_81c9387d2e7576da065747ec9c16d1c7d69016ec/qa/tasks 2023-07-14T18:35:48.897 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-07-14T18:35:48.899 INFO:teuthology.task.internal:Checking packages... 2023-07-14T18:35:48.929 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '81c9387d2e7576da065747ec9c16d1c7d69016ec' 2023-07-14T18:35:48.929 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-07-14T18:35:48.930 INFO:teuthology.packaging:ref: None 2023-07-14T18:35:48.930 INFO:teuthology.packaging:tag: None 2023-07-14T18:35:48.930 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230714.134335 2023-07-14T18:35:48.930 INFO:teuthology.packaging:sha1: 81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:35:48.931 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230714.134335 2023-07-14T18:35:49.237 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-4940.g81c9387d 2023-07-14T18:35:49.237 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-07-14T18:35:49.242 INFO:teuthology.task.internal:no buildpackages task found 2023-07-14T18:35:49.242 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-07-14T18:35:49.272 INFO:teuthology.task.internal:Saving configuration 2023-07-14T18:35:49.293 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-07-14T18:35:49.299 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T18:35:54.026 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:27:54.604030', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:35:54.050 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:27:54.602748', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:35:54.072 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:27:54.601276', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T18:35:54.072 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-07-14T18:35:54.081 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2023-07-14T18:35:54.081 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2023-07-14T18:35:54.082 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2023-07-14T18:35:54.082 INFO:teuthology.run_tasks:Running task console_log... 2023-07-14T18:35:54.198 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5474de93a0>, signals=[15]) 2023-07-14T18:35:54.199 INFO:teuthology.run_tasks:Running task internal.connect... 2023-07-14T18:35:54.205 INFO:teuthology.task.internal:Opening connections... 2023-07-14T18:35:54.205 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2023-07-14T18:35:54.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:35:54.278 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T18:35:54.280 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:35:54.347 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2023-07-14T18:35:54.348 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:35:54.415 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-07-14T18:35:54.425 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2023-07-14T18:35:54.444 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2023-07-14T18:35:54.444 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2023-07-14T18:35:54.502 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:35:54.502 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:35:54.502 INFO:teuthology.orchestra.run.smithi078.stdout:ID="rhel" 2023-07-14T18:35:54.503 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE="fedora" 2023-07-14T18:35:54.503 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="8.6" 2023-07-14T18:35:54.503 INFO:teuthology.orchestra.run.smithi078.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:35:54.503 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:35:54.503 INFO:teuthology.orchestra.run.smithi078.stdout:ANSI_COLOR="0;31" 2023-07-14T18:35:54.503 INFO:teuthology.orchestra.run.smithi078.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:35:54.503 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:35:54.504 INFO:teuthology.orchestra.run.smithi078.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:35:54.504 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:35:54.504 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:35:54.504 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:35:54.504 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:35:54.504 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:35:54.504 INFO:teuthology.orchestra.run.smithi078.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:35:54.505 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2023-07-14T18:35:54.535 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2023-07-14T18:35:54.554 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2023-07-14T18:35:54.555 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2023-07-14T18:35:54.613 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:35:54.613 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:35:54.613 INFO:teuthology.orchestra.run.smithi088.stdout:ID="rhel" 2023-07-14T18:35:54.614 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="fedora" 2023-07-14T18:35:54.614 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="8.6" 2023-07-14T18:35:54.614 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:35:54.614 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:35:54.614 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2023-07-14T18:35:54.615 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:35:54.615 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:35:54.615 INFO:teuthology.orchestra.run.smithi088.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:35:54.615 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:35:54.615 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:35:54.616 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:35:54.616 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:35:54.616 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:35:54.616 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:35:54.616 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2023-07-14T18:35:54.643 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2023-07-14T18:35:54.662 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2023-07-14T18:35:54.663 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2023-07-14T18:35:54.720 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="Red Hat Enterprise Linux" 2023-07-14T18:35:54.720 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="8.6 (Ootpa)" 2023-07-14T18:35:54.720 INFO:teuthology.orchestra.run.smithi102.stdout:ID="rhel" 2023-07-14T18:35:54.721 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="fedora" 2023-07-14T18:35:54.721 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="8.6" 2023-07-14T18:35:54.721 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el8" 2023-07-14T18:35:54.721 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-07-14T18:35:54.721 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2023-07-14T18:35:54.721 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-07-14T18:35:54.721 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://www.redhat.com/" 2023-07-14T18:35:54.722 INFO:teuthology.orchestra.run.smithi102.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-07-14T18:35:54.722 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-07-14T18:35:54.722 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:35:54.722 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-07-14T18:35:54.722 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-07-14T18:35:54.722 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-07-14T18:35:54.722 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-07-14T18:35:54.723 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2023-07-14T18:35:54.744 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-07-14T18:35:54.752 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-07-14T18:35:54.761 INFO:teuthology.task.internal:Checking for old test directory... 2023-07-14T18:35:54.761 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:35:54.763 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:35:54.766 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2023-07-14T18:35:54.782 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-07-14T18:35:54.789 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-07-14T18:35:54.789 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:35:54.824 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:35:54.826 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2023-07-14T18:35:54.852 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-07-14T18:35:54.939 INFO:teuthology.run_tasks:Running task kernel... 2023-07-14T18:35:54.953 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-07-14T18:35:54.954 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-07-14T18:35:54.954 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:35:54.955 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:35:54.956 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2023-07-14T18:35:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:35:54.974 DEBUG:teuthology.orchestra.run.smithi102:> uname -r 2023-07-14T18:35:54.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:35:54.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:35:54.976 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2023-07-14T18:35:54.976 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2023-07-14T18:35:55.032 INFO:teuthology.orchestra.run.smithi102.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T18:35:55.035 INFO:teuthology.task.kernel:Running kernel on smithi102: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:35:55.035 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum install -y kernel 2023-07-14T18:35:55.038 INFO:teuthology.orchestra.run.smithi078.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T18:35:55.038 INFO:teuthology.orchestra.run.smithi088.stdout:4.18.0-372.9.1.el8.x86_64 2023-07-14T18:35:55.038 INFO:teuthology.task.kernel:Running kernel on smithi078: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:35:55.039 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum install -y kernel 2023-07-14T18:35:55.039 INFO:teuthology.task.kernel:Running kernel on smithi088: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:35:55.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2023-07-14T18:35:56.228 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2023-07-14T18:35:56.228 INFO:teuthology.orchestra.run.smithi102.stdout:Unable to read consumer identity 2023-07-14T18:35:56.299 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-07-14T18:35:56.299 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-07-14T18:35:56.329 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T18:35:56.329 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-07-14T18:35:57.632 INFO:teuthology.orchestra.run.smithi102.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-07-14T18:35:57.751 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-07-14T18:35:57.855 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-07-14T18:36:01.457 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 4.1 MB/s | 16 MB 00:03 2023-07-14T18:36:02.289 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 3.5 MB/s | 16 MB 00:04 2023-07-14T18:36:02.835 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 3.1 MB/s | 16 MB 00:05 2023-07-14T18:36:04.941 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 439 kB/s | 24 kB 00:00 2023-07-14T18:36:05.610 INFO:teuthology.orchestra.run.smithi102.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T18:36:05.671 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2023-07-14T18:36:05.672 INFO:teuthology.orchestra.run.smithi102.stdout:Nothing to do. 2023-07-14T18:36:05.672 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2023-07-14T18:36:05.689 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 448 kB/s | 24 kB 00:00 2023-07-14T18:36:05.804 DEBUG:teuthology.orchestra.run.smithi102:> echo no | sudo yum reinstall kernel || true 2023-07-14T18:36:06.134 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2023-07-14T18:36:06.134 INFO:teuthology.orchestra.run.smithi102.stdout:Unable to read consumer identity 2023-07-14T18:36:06.377 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 439 kB/s | 24 kB 00:00 2023-07-14T18:36:06.382 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T18:36:06.396 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 06:36:04 PM UTC. 2023-07-14T18:36:06.435 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-14T18:36:06.436 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2023-07-14T18:36:06.437 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-14T18:36:06.439 INFO:teuthology.orchestra.run.smithi102.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:36:06.468 INFO:teuthology.orchestra.run.smithi102.stderr:Error: No packages marked for reinstall. 2023-07-14T18:36:06.514 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum reinstall -y kernel || true 2023-07-14T18:36:06.562 DEBUG:teuthology.orchestra.run.smithi088:> echo no | sudo yum reinstall kernel || true 2023-07-14T18:36:06.842 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2023-07-14T18:36:06.842 INFO:teuthology.orchestra.run.smithi102.stdout:Unable to read consumer identity 2023-07-14T18:36:06.893 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T18:36:06.893 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-07-14T18:36:07.054 INFO:teuthology.orchestra.run.smithi078.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-07-14T18:36:07.100 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:00:03 ago on Fri 14 Jul 2023 06:36:04 PM UTC. 2023-07-14T18:36:07.104 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-07-14T18:36:07.105 INFO:teuthology.orchestra.run.smithi078.stdout:Nothing to do. 2023-07-14T18:36:07.105 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-07-14T18:36:07.140 INFO:teuthology.orchestra.run.smithi102.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:36:07.157 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 06:36:05 PM UTC. 2023-07-14T18:36:07.168 INFO:teuthology.orchestra.run.smithi102.stderr:Error: No packages marked for reinstall. 2023-07-14T18:36:07.197 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:36:07.218 DEBUG:teuthology.orchestra.run.smithi078:> echo no | sudo yum reinstall kernel || true 2023-07-14T18:36:07.219 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T18:36:07.236 INFO:teuthology.orchestra.run.smithi088.stderr:Error: No packages marked for reinstall. 2023-07-14T18:36:07.281 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum reinstall -y kernel || true 2023-07-14T18:36:07.299 INFO:teuthology.orchestra.run.smithi102.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T18:36:07.299 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:36:07.299 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T18:36:07.299 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-07-14T18:36:07.300 DEBUG:teuthology.orchestra.run.smithi102:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T18:36:07.332 INFO:teuthology.orchestra.run.smithi102.stdout:ttyS1 2023-07-14T18:36:07.344 DEBUG:teuthology.parallel:result is None 2023-07-14T18:36:07.549 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-07-14T18:36:07.550 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-07-14T18:36:07.612 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T18:36:07.612 INFO:teuthology.orchestra.run.smithi088.stdout:Unable to read consumer identity 2023-07-14T18:36:07.809 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:01 ago on Fri 14 Jul 2023 06:36:06 PM UTC. 2023-07-14T18:36:07.861 INFO:teuthology.orchestra.run.smithi078.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:36:07.867 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 06:36:05 PM UTC. 2023-07-14T18:36:07.889 INFO:teuthology.orchestra.run.smithi078.stderr:Error: No packages marked for reinstall. 2023-07-14T18:36:07.907 INFO:teuthology.orchestra.run.smithi088.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:36:07.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum reinstall -y kernel || true 2023-07-14T18:36:07.943 INFO:teuthology.orchestra.run.smithi088.stderr:Error: No packages marked for reinstall. 2023-07-14T18:36:07.988 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T18:36:08.067 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T18:36:08.068 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:36:08.069 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T18:36:08.069 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-07-14T18:36:08.070 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T18:36:08.103 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2023-07-14T18:36:08.122 DEBUG:teuthology.parallel:result is None 2023-07-14T18:36:08.263 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-07-14T18:36:08.264 INFO:teuthology.orchestra.run.smithi078.stdout:Unable to read consumer identity 2023-07-14T18:36:08.520 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:00:02 ago on Fri 14 Jul 2023 06:36:06 PM UTC. 2023-07-14T18:36:08.562 INFO:teuthology.orchestra.run.smithi078.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-07-14T18:36:08.587 INFO:teuthology.orchestra.run.smithi078.stderr:Error: No packages marked for reinstall. 2023-07-14T18:36:08.638 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q kernel | sort -rV | head -n 1 2023-07-14T18:36:08.718 INFO:teuthology.orchestra.run.smithi078.stdout:kernel-4.18.0-372.9.1.el8.x86_64 2023-07-14T18:36:08.719 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-07-14T18:36:08.719 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-07-14T18:36:08.719 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-07-14T18:36:08.719 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-07-14T18:36:08.752 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2023-07-14T18:36:08.771 DEBUG:teuthology.parallel:result is None 2023-07-14T18:36:08.772 INFO:teuthology.run_tasks:Running task internal.base... 2023-07-14T18:36:08.783 INFO:teuthology.task.internal:Creating test directory... 2023-07-14T18:36:08.783 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:36:08.814 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:36:08.816 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-07-14T18:36:08.836 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-07-14T18:36:08.843 INFO:teuthology.run_tasks:Running task internal.archive... 2023-07-14T18:36:08.859 INFO:teuthology.task.internal:Creating archive directory... 2023-07-14T18:36:08.859 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:36:08.869 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:36:08.876 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-07-14T18:36:08.903 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-07-14T18:36:08.915 INFO:teuthology.task.internal:Enabling coredump saving... 2023-07-14T18:36:08.915 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:36:08.931 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:36:08.936 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-07-14T18:36:08.972 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:36:08.979 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:36:08.984 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:36:08.991 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:36:09.006 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:36:09.016 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-07-14T18:36:09.017 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-07-14T18:36:09.028 INFO:teuthology.task.internal:Configuring sudo... 2023-07-14T18:36:09.028 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:36:09.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:36:09.033 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-07-14T18:36:09.065 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-07-14T18:36:09.074 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-07-14T18:36:09.075 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:36:09.097 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:36:09.102 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-07-14T18:36:09.124 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:36:09.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:36:09.254 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:36:09.301 DEBUG:teuthology.orchestra.run.smithi078:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:36:09.337 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:36:09.338 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:36:09.405 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:36:09.454 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:36:09.503 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:36:14.594 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:36:14.626 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:36:14.627 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:36:14.698 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:36:24.752 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-07-14T18:36:24.822 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:36:34.875 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-07-14T18:36:34.906 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:36:34.906 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-07-14T18:36:34.976 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2023-07-14T18:36:34.978 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2023-07-14T18:36:34.981 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2023-07-14T18:36:35.054 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:36:35.057 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:36:35.097 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T18:36:35.486 INFO:teuthology.run_tasks:Running task internal.timer... 2023-07-14T18:36:35.494 INFO:teuthology.task.internal:Starting timer... 2023-07-14T18:36:35.494 INFO:teuthology.run_tasks:Running task pcp... 2023-07-14T18:36:35.507 INFO:teuthology.run_tasks:Running task selinux... 2023-07-14T18:36:35.518 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-07-14T18:36:35.519 DEBUG:teuthology.orchestra.run.smithi078:> sudo service auditd rotate 2023-07-14T18:36:35.588 INFO:teuthology.orchestra.run.smithi078.stdout:Rotating logs: 2023-07-14T18:36:35.590 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2023-07-14T18:36:35.647 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2023-07-14T18:36:35.648 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2023-07-14T18:36:35.716 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2023-07-14T18:36:35.718 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-07-14T18:36:35.718 DEBUG:teuthology.orchestra.run.smithi078:> /usr/sbin/getenforce 2023-07-14T18:36:35.740 INFO:teuthology.orchestra.run.smithi078.stdout:Permissive 2023-07-14T18:36:35.740 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2023-07-14T18:36:35.764 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2023-07-14T18:36:35.765 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2023-07-14T18:36:35.794 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2023-07-14T18:36:35.794 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi078.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi102.front.sepia.ceph.com': 'permissive'} 2023-07-14T18:36:35.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:36:35.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:36:35.822 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:36:35.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:36:35.850 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T18:36:35.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T18:36:35.877 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-07-14T18:36:35.877 DEBUG:teuthology.orchestra.run.smithi078:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:36:35.905 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:36:35.942 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2023-07-14T18:36:35.968 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-07-14T18:36:35.978 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-07-14T18:36:35.984 INFO:teuthology.repo_utils:Fetching main from origin 2023-07-14T18:36:41.137 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-07-14T18:36:41.157 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-07-14T18:36:41.159 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi088.front.sepia.ceph.com,smithi102.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-07-14T18:50:59.968 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi102.front.sepia.ceph.com')] 2023-07-14T18:50:59.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-14T18:50:59.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:51:05.043 DEBUG:teuthology.orchestra.run.smithi078:> true 2023-07-14T18:51:05.116 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2023-07-14T18:51:05.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-07-14T18:51:05.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:51:10.187 DEBUG:teuthology.orchestra.run.smithi088:> true 2023-07-14T18:51:10.256 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2023-07-14T18:51:10.256 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2023-07-14T18:51:10.257 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-07-14T18:51:15.329 DEBUG:teuthology.orchestra.run.smithi102:> true 2023-07-14T18:51:15.400 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2023-07-14T18:51:15.401 INFO:teuthology.run_tasks:Running task clock... 2023-07-14T18:51:15.411 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-07-14T18:51:15.412 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T18:51:15.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:51:15.415 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T18:51:15.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:51:15.418 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-07-14T18:51:15.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T18:51:15.450 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T18:51:15.452 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T18:51:15.466 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T18:51:15.468 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T18:51:15.476 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-07-14T18:51:15.492 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-07-14T18:51:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2023-07-14T18:51:15.544 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2023-07-14T18:51:15.547 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: ntpd: command not found 2023-07-14T18:51:15.559 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T18:51:15.560 INFO:teuthology.orchestra.run.smithi078.stdout:506 Cannot talk to daemon 2023-07-14T18:51:15.574 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T18:51:15.575 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T18:51:15.577 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2023-07-14T18:51:15.589 INFO:teuthology.orchestra.run.smithi078.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T18:51:15.590 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2023-07-14T18:51:15.604 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-07-14T18:51:15.619 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-07-14T18:51:15.639 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2023-07-14T18:51:15.642 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:51:15.642 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-07-14T18:51:15.643 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.643 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.643 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.643 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.661 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2023-07-14T18:51:15.664 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:51:15.664 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2023-07-14T18:51:15.664 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.664 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.665 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.665 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.693 INFO:teuthology.orchestra.run.smithi102.stderr:bash: ntpq: command not found 2023-07-14T18:51:15.697 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T18:51:15.698 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2023-07-14T18:51:15.698 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.698 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.698 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.698 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T18:51:15.698 INFO:teuthology.run_tasks:Running task pexec... 2023-07-14T18:51:15.708 INFO:teuthology.task.pexec:Executing custom commands... 2023-07-14T18:51:15.708 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi078.front.sepia.ceph.com 2023-07-14T18:51:15.709 DEBUG:teuthology.orchestra.run.smithi078:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T18:51:15.709 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T18:51:15.710 DEBUG:teuthology.orchestra.run.smithi088:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T18:51:15.710 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi102.front.sepia.ceph.com 2023-07-14T18:51:15.710 DEBUG:teuthology.orchestra.run.smithi102:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-07-14T18:51:16.080 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:16.083 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:16.110 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:17.105 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:33 ago on Fri 14 Jul 2023 06:44:43 PM UTC. 2023-07-14T18:51:17.227 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:40 ago on Fri 14 Jul 2023 06:44:36 PM UTC. 2023-07-14T18:51:17.248 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:23 ago on Fri 14 Jul 2023 06:48:53 PM UTC. 2023-07-14T18:51:18.823 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-14T18:51:18.825 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-14T18:51:18.825 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2023-07-14T18:51:18.825 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-14T18:51:18.825 INFO:teuthology.orchestra.run.smithi088.stdout:Resetting modules: 2023-07-14T18:51:18.825 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools 2023-07-14T18:51:18.825 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:51:18.825 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-14T18:51:18.826 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2023-07-14T18:51:18.826 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:51:18.949 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2023-07-14T18:51:18.950 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2023-07-14T18:51:18.950 INFO:teuthology.orchestra.run.smithi102.stdout: Package Architecture Version Repository Size 2023-07-14T18:51:18.950 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2023-07-14T18:51:18.950 INFO:teuthology.orchestra.run.smithi102.stdout:Resetting modules: 2023-07-14T18:51:18.951 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools 2023-07-14T18:51:18.951 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:51:18.951 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2023-07-14T18:51:18.951 INFO:teuthology.orchestra.run.smithi102.stdout:================================================================================ 2023-07-14T18:51:18.951 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:51:18.970 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-07-14T18:51:18.971 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-07-14T18:51:18.971 INFO:teuthology.orchestra.run.smithi078.stdout: Package Architecture Version Repository Size 2023-07-14T18:51:18.971 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-07-14T18:51:18.971 INFO:teuthology.orchestra.run.smithi078.stdout:Resetting modules: 2023-07-14T18:51:18.971 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools 2023-07-14T18:51:18.971 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:51:18.972 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-07-14T18:51:18.972 INFO:teuthology.orchestra.run.smithi078.stdout:================================================================================ 2023-07-14T18:51:18.972 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:51:19.774 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:36 ago on Fri 14 Jul 2023 06:44:43 PM UTC. 2023-07-14T18:51:19.881 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:43 ago on Fri 14 Jul 2023 06:44:36 PM UTC. 2023-07-14T18:51:19.934 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:26 ago on Fri 14 Jul 2023 06:48:53 PM UTC. 2023-07-14T18:51:25.933 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2023-07-14T18:51:26.061 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-14T18:51:26.476 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-07-14T18:51:26.506 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:26.676 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:26.945 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:28.151 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:44 ago on Fri 14 Jul 2023 06:44:43 PM UTC. 2023-07-14T18:51:28.292 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:06:51 ago on Fri 14 Jul 2023 06:44:36 PM UTC. 2023-07-14T18:51:28.298 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:34 ago on Fri 14 Jul 2023 06:48:53 PM UTC. 2023-07-14T18:51:29.892 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-14T18:51:29.984 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-14T18:51:29.985 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-07-14T18:51:29.985 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-14T18:51:29.985 INFO:teuthology.orchestra.run.smithi088.stdout:Installing group/module packages: 2023-07-14T18:51:29.985 INFO:teuthology.orchestra.run.smithi088.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T18:51:29.985 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T18:51:29.985 INFO:teuthology.orchestra.run.smithi088.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T18:51:29.985 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T18:51:29.986 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T18:51:29.986 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T18:51:29.986 INFO:teuthology.orchestra.run.smithi088.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T18:51:29.986 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-07-14T18:51:29.986 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T18:51:29.986 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T18:51:29.986 INFO:teuthology.orchestra.run.smithi088.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T18:51:29.986 INFO:teuthology.orchestra.run.smithi088.stdout:Installing module profiles: 2023-07-14T18:51:29.987 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools/common 2023-07-14T18:51:29.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:51:29.987 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling module streams: 2023-07-14T18:51:29.987 INFO:teuthology.orchestra.run.smithi088.stdout: container-tools rhel8 2023-07-14T18:51:29.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:51:29.987 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-14T18:51:29.987 INFO:teuthology.orchestra.run.smithi088.stdout:============================================================================================================ 2023-07-14T18:51:29.988 INFO:teuthology.orchestra.run.smithi088.stdout:Install 10 Packages 2023-07-14T18:51:29.988 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:51:29.988 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 18 M 2023-07-14T18:51:29.988 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 65 M 2023-07-14T18:51:29.989 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-07-14T18:51:30.021 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-07-14T18:51:30.073 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2023-07-14T18:51:30.113 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================================ 2023-07-14T18:51:30.113 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-07-14T18:51:30.113 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================================ 2023-07-14T18:51:30.113 INFO:teuthology.orchestra.run.smithi078.stdout:Installing group/module packages: 2023-07-14T18:51:30.114 INFO:teuthology.orchestra.run.smithi078.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T18:51:30.114 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T18:51:30.114 INFO:teuthology.orchestra.run.smithi078.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T18:51:30.114 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T18:51:30.114 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T18:51:30.114 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T18:51:30.114 INFO:teuthology.orchestra.run.smithi078.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T18:51:30.115 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-07-14T18:51:30.115 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T18:51:30.115 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T18:51:30.115 INFO:teuthology.orchestra.run.smithi078.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T18:51:30.115 INFO:teuthology.orchestra.run.smithi078.stdout:Installing module profiles: 2023-07-14T18:51:30.115 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools/common 2023-07-14T18:51:30.115 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:51:30.115 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2023-07-14T18:51:30.116 INFO:teuthology.orchestra.run.smithi078.stdout: container-tools rhel8 2023-07-14T18:51:30.116 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:51:30.116 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-07-14T18:51:30.116 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================================================ 2023-07-14T18:51:30.116 INFO:teuthology.orchestra.run.smithi078.stdout:Install 10 Packages 2023-07-14T18:51:30.116 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:51:30.117 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 18 M 2023-07-14T18:51:30.117 INFO:teuthology.orchestra.run.smithi078.stdout:Installed size: 65 M 2023-07-14T18:51:30.117 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-07-14T18:51:30.124 INFO:teuthology.orchestra.run.smithi088.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 298 kB/s | 41 kB 00:00 2023-07-14T18:51:30.165 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================================ 2023-07-14T18:51:30.165 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2023-07-14T18:51:30.165 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================================ 2023-07-14T18:51:30.165 INFO:teuthology.orchestra.run.smithi102.stdout:Installing group/module packages: 2023-07-14T18:51:30.165 INFO:teuthology.orchestra.run.smithi102.stdout: buildah x86_64 1:1.26.2-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 8.0 M 2023-07-14T18:51:30.165 INFO:teuthology.orchestra.run.smithi102.stdout: cockpit-podman noarch 49.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 524 k 2023-07-14T18:51:30.166 INFO:teuthology.orchestra.run.smithi102.stdout: crun x86_64 1.5-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 213 k 2023-07-14T18:51:30.166 INFO:teuthology.orchestra.run.smithi102.stdout: python3-podman noarch 4.0.1-1.module+el8.6.0+16837+85b82211 rhel-8-for-x86_64-appstream-rpms 149 k 2023-07-14T18:51:30.166 INFO:teuthology.orchestra.run.smithi102.stdout: skopeo x86_64 2:1.9.1-1.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 7.0 M 2023-07-14T18:51:30.166 INFO:teuthology.orchestra.run.smithi102.stdout: toolbox x86_64 0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 2.2 M 2023-07-14T18:51:30.166 INFO:teuthology.orchestra.run.smithi102.stdout: udica noarch 0.2.6-4.module+el8.6.0+16771+28dfca77 rhel-8-for-x86_64-appstream-rpms 49 k 2023-07-14T18:51:30.166 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2023-07-14T18:51:30.166 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 rhel-8-for-x86_64-appstream-rpms 25 k 2023-07-14T18:51:30.166 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyxdg noarch 0.25-16.el8 rhel-8-for-x86_64-appstream-rpms 94 k 2023-07-14T18:51:30.167 INFO:teuthology.orchestra.run.smithi102.stdout: yajl x86_64 2.1.0-10.el8 rhel-8-for-x86_64-appstream-rpms 41 k 2023-07-14T18:51:30.167 INFO:teuthology.orchestra.run.smithi102.stdout:Installing module profiles: 2023-07-14T18:51:30.167 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools/common 2023-07-14T18:51:30.167 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:51:30.167 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling module streams: 2023-07-14T18:51:30.167 INFO:teuthology.orchestra.run.smithi102.stdout: container-tools rhel8 2023-07-14T18:51:30.167 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:51:30.167 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2023-07-14T18:51:30.168 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================================ 2023-07-14T18:51:30.168 INFO:teuthology.orchestra.run.smithi102.stdout:Install 10 Packages 2023-07-14T18:51:30.168 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:51:30.168 INFO:teuthology.orchestra.run.smithi088.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 525 kB/s | 94 kB 00:00 2023-07-14T18:51:30.169 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 18 M 2023-07-14T18:51:30.169 INFO:teuthology.orchestra.run.smithi102.stdout:Installed size: 65 M 2023-07-14T18:51:30.169 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2023-07-14T18:51:30.191 INFO:teuthology.orchestra.run.smithi088.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 122 kB/s | 25 kB 00:00 2023-07-14T18:51:30.216 INFO:teuthology.orchestra.run.smithi088.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 536 kB/s | 49 kB 00:00 2023-07-14T18:51:30.235 INFO:teuthology.orchestra.run.smithi078.stdout:(1/10): yajl-2.1.0-10.el8.x86_64.rpm 343 kB/s | 41 kB 00:00 2023-07-14T18:51:30.293 INFO:teuthology.orchestra.run.smithi078.stdout:(2/10): python3-pyxdg-0.25-16.el8.noarch.rpm 529 kB/s | 94 kB 00:00 2023-07-14T18:51:30.308 INFO:teuthology.orchestra.run.smithi088.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.6 MB/s | 524 kB 00:00 2023-07-14T18:51:30.310 INFO:teuthology.orchestra.run.smithi078.stdout:(3/10): python3-pytoml-0.1.14-5.git7dea353.el8. 128 kB/s | 25 kB 00:00 2023-07-14T18:51:30.344 INFO:teuthology.orchestra.run.smithi078.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 453 kB/s | 49 kB 00:00 2023-07-14T18:51:30.350 INFO:teuthology.orchestra.run.smithi088.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.1 MB/s | 149 kB 00:00 2023-07-14T18:51:30.368 INFO:teuthology.orchestra.run.smithi102.stdout:(1/10): python3-pytoml-0.1.14-5.git7dea353.el8. 123 kB/s | 25 kB 00:00 2023-07-14T18:51:30.394 INFO:teuthology.orchestra.run.smithi102.stdout:(2/10): yajl-2.1.0-10.el8.x86_64.rpm 180 kB/s | 41 kB 00:00 2023-07-14T18:51:30.411 INFO:teuthology.orchestra.run.smithi102.stdout:(3/10): python3-pyxdg-0.25-16.el8.noarch.rpm 385 kB/s | 94 kB 00:00 2023-07-14T18:51:30.452 INFO:teuthology.orchestra.run.smithi078.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 3.2 MB/s | 524 kB 00:00 2023-07-14T18:51:30.459 INFO:teuthology.orchestra.run.smithi088.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 30 MB/s | 8.0 MB 00:00 2023-07-14T18:51:30.484 INFO:teuthology.orchestra.run.smithi088.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.2 MB/s | 213 kB 00:00 2023-07-14T18:51:30.544 INFO:teuthology.orchestra.run.smithi102.stdout:(4/10): udica-0.2.6-4.module+el8.6.0+16771+28df 281 kB/s | 49 kB 00:00 2023-07-14T18:51:30.570 INFO:teuthology.orchestra.run.smithi078.stdout:(6/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 1.8 MB/s | 213 kB 00:00 2023-07-14T18:51:30.595 INFO:teuthology.orchestra.run.smithi102.stdout:(5/10): cockpit-podman-49.1-1.module+el8.6.0+16 2.6 MB/s | 524 kB 00:00 2023-07-14T18:51:30.609 INFO:teuthology.orchestra.run.smithi088.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 27 MB/s | 7.0 MB 00:00 2023-07-14T18:51:30.628 INFO:teuthology.orchestra.run.smithi078.stdout:(7/10): python3-podman-4.0.1-1.module+el8.6.0+1 525 kB/s | 149 kB 00:00 2023-07-14T18:51:30.670 INFO:teuthology.orchestra.run.smithi102.stdout:(6/10): python3-podman-4.0.1-1.module+el8.6.0+1 1.2 MB/s | 149 kB 00:00 2023-07-14T18:51:30.684 INFO:teuthology.orchestra.run.smithi088.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 9.7 MB/s | 2.2 MB 00:00 2023-07-14T18:51:30.685 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:51:30.685 INFO:teuthology.orchestra.run.smithi088.stdout:Total 26 MB/s | 18 MB 00:00 2023-07-14T18:51:30.779 INFO:teuthology.orchestra.run.smithi078.stdout:(8/10): buildah-1.26.2-1.module+el8.6.0+16771+2 17 MB/s | 8.0 MB 00:00 2023-07-14T18:51:30.787 INFO:teuthology.orchestra.run.smithi102.stdout:(7/10): buildah-1.26.2-1.module+el8.6.0+16771+2 21 MB/s | 8.0 MB 00:00 2023-07-14T18:51:30.812 INFO:teuthology.orchestra.run.smithi102.stdout:(8/10): crun-1.5-1.module+el8.6.0+16771+28dfca7 981 kB/s | 213 kB 00:00 2023-07-14T18:51:30.845 INFO:teuthology.orchestra.run.smithi078.stdout:(9/10): toolbox-0.0.99.3-0.6.module+el8.6.0+167 10 MB/s | 2.2 MB 00:00 2023-07-14T18:51:30.886 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-07-14T18:51:30.905 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-07-14T18:51:30.905 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-07-14T18:51:30.937 INFO:teuthology.orchestra.run.smithi078.stdout:(10/10): skopeo-1.9.1-1.module+el8.6.0+16771+28 19 MB/s | 7.0 MB 00:00 2023-07-14T18:51:30.938 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:51:30.938 INFO:teuthology.orchestra.run.smithi078.stdout:Total 22 MB/s | 18 MB 00:00 2023-07-14T18:51:30.939 INFO:teuthology.orchestra.run.smithi102.stdout:(9/10): skopeo-1.9.1-1.module+el8.6.0+16771+28d 26 MB/s | 7.0 MB 00:00 2023-07-14T18:51:30.998 INFO:teuthology.orchestra.run.smithi102.stdout:(10/10): toolbox-0.0.99.3-0.6.module+el8.6.0+16 10 MB/s | 2.2 MB 00:00 2023-07-14T18:51:30.999 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:51:30.999 INFO:teuthology.orchestra.run.smithi102.stdout:Total 22 MB/s | 18 MB 00:00 2023-07-14T18:51:31.086 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-07-14T18:51:31.090 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-07-14T18:51:31.137 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-07-14T18:51:31.155 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-07-14T18:51:31.156 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-07-14T18:51:31.197 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2023-07-14T18:51:31.216 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2023-07-14T18:51:31.216 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2023-07-14T18:51:31.336 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-07-14T18:51:31.340 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-07-14T18:51:31.398 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2023-07-14T18:51:31.402 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2023-07-14T18:51:31.621 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-07-14T18:51:31.824 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T18:51:31.860 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-07-14T18:51:31.929 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2023-07-14T18:51:31.977 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:51:32.075 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T18:51:32.122 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 1/10 2023-07-14T18:51:32.185 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T18:51:32.252 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:51:32.295 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:51:32.403 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T18:51:32.462 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T18:51:32.489 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 3/10 2023-07-14T18:51:32.689 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T18:51:32.700 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 4/10 2023-07-14T18:51:32.805 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T18:51:33.038 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T18:51:33.073 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 5/10 2023-07-14T18:51:33.600 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T18:51:33.783 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T18:51:33.851 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 6/10 2023-07-14T18:51:34.522 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T18:51:34.707 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T18:51:34.712 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 7/10 2023-07-14T18:51:34.782 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T18:51:34.931 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T18:51:34.959 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T18:51:34.976 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 8/10 2023-07-14T18:51:35.093 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T18:51:35.159 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 9/10 2023-07-14T18:51:35.169 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:51:35.288 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:51:35.369 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:51:35.436 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:51:35.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T18:51:35.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:51:35.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T18:51:35.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T18:51:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T18:51:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T18:51:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T18:51:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T18:51:35.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T18:51:35.549 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:51:35.549 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T18:51:35.550 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:51:35.550 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T18:51:35.550 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T18:51:35.550 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T18:51:35.550 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T18:51:35.550 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T18:51:35.550 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T18:51:35.551 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T18:51:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 10/10 2023-07-14T18:51:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 1/10 2023-07-14T18:51:35.631 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/10 2023-07-14T18:51:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/10 2023-07-14T18:51:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 4/10 2023-07-14T18:51:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca7 5/10 2023-07-14T18:51:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x 6/10 2023-07-14T18:51:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-podman-4.0.1-1.module+el8.6.0+16837+85b822 7/10 2023-07-14T18:51:35.632 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 8/10 2023-07-14T18:51:35.633 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86 9/10 2023-07-14T18:51:36.669 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T18:51:36.669 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:53 ago on Fri 14 Jul 2023 06:44:43 PM UTC. 2023-07-14T18:51:36.778 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T18:51:36.778 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:07:00 ago on Fri 14 Jul 2023 06:44:36 PM UTC. 2023-07-14T18:51:36.792 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77 10/10 2023-07-14T18:51:36.792 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:43 ago on Fri 14 Jul 2023 06:48:53 PM UTC. 2023-07-14T18:51:42.462 INFO:teuthology.orchestra.run.smithi088.stdout:Installed products updated. 2023-07-14T18:51:42.668 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:51:42.668 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-07-14T18:51:42.668 INFO:teuthology.orchestra.run.smithi088.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:42.668 INFO:teuthology.orchestra.run.smithi088.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:51:42.668 INFO:teuthology.orchestra.run.smithi088.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:42.668 INFO:teuthology.orchestra.run.smithi088.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T18:51:42.669 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T18:51:42.669 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T18:51:42.669 INFO:teuthology.orchestra.run.smithi088.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:42.669 INFO:teuthology.orchestra.run.smithi088.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:42.669 INFO:teuthology.orchestra.run.smithi088.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:51:42.669 INFO:teuthology.orchestra.run.smithi088.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T18:51:42.669 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:51:42.669 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-14T18:51:42.825 DEBUG:teuthology.parallel:result is None 2023-07-14T18:51:43.936 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2023-07-14T18:51:44.132 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:51:44.133 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2023-07-14T18:51:44.133 INFO:teuthology.orchestra.run.smithi102.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:44.133 INFO:teuthology.orchestra.run.smithi102.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:51:44.133 INFO:teuthology.orchestra.run.smithi102.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:44.133 INFO:teuthology.orchestra.run.smithi102.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T18:51:44.133 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T18:51:44.133 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T18:51:44.134 INFO:teuthology.orchestra.run.smithi102.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:44.134 INFO:teuthology.orchestra.run.smithi102.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:44.134 INFO:teuthology.orchestra.run.smithi102.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:51:44.134 INFO:teuthology.orchestra.run.smithi102.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T18:51:44.134 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:51:44.134 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2023-07-14T18:51:44.279 DEBUG:teuthology.parallel:result is None 2023-07-14T18:51:44.624 INFO:teuthology.orchestra.run.smithi078.stdout:Installed products updated. 2023-07-14T18:51:44.857 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:51:44.857 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-07-14T18:51:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: buildah-1:1.26.2-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: cockpit-podman-49.1-1.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:51:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: crun-1.5-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: python3-podman-4.0.1-1.module+el8.6.0+16837+85b82211.noarch 2023-07-14T18:51:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-07-14T18:51:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-07-14T18:51:44.858 INFO:teuthology.orchestra.run.smithi078.stdout: skopeo-2:1.9.1-1.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: toolbox-0.0.99.3-0.6.module+el8.6.0+16771+28dfca77.x86_64 2023-07-14T18:51:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: udica-0.2.6-4.module+el8.6.0+16771+28dfca77.noarch 2023-07-14T18:51:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: yajl-2.1.0-10.el8.x86_64 2023-07-14T18:51:44.859 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:51:44.859 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-07-14T18:51:45.016 DEBUG:teuthology.parallel:result is None 2023-07-14T18:51:45.017 INFO:teuthology.run_tasks:Running task install... 2023-07-14T18:51:45.026 DEBUG:teuthology.task.install:project ceph 2023-07-14T18:51:45.027 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'}} 2023-07-14T18:51:45.027 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec'} 2023-07-14T18:51:45.028 INFO:teuthology.task.install:Using flavor: default 2023-07-14T18:51:45.034 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-07-14T18:51:45.034 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-07-14T18:51:45.035 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T18:51:45.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:51:45.037 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T18:51:45.038 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:51:45.039 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'tag': None, 'wait_for_package': False} 2023-07-14T18:51:45.039 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:51:47.207 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T18:51:47.207 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T18:51:47.213 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T18:51:47.213 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T18:51:47.264 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/ 2023-07-14T18:51:47.265 INFO:teuthology.task.install.rpm:Package version is 18.0.0-4940.g81c9387d 2023-07-14T18:51:52.450 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T18:51:52.450 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:51:52.450 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T18:51:52.453 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T18:51:52.453 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:51:52.453 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T18:51:52.486 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T18:51:52.486 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T18:51:52.487 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T18:51:52.488 DEBUG:teuthology.orchestra.run.smithi078:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T18:51:52.556 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T18:51:52.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T18:51:52.634 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-07-14T18:51:52.634 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:51:52.634 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-07-14T18:51:52.637 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T18:51:52.638 DEBUG:teuthology.orchestra.run.smithi078:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T18:51:52.667 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-07-14T18:51:52.668 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/81c9387d2e7576da065747ec9c16d1c7d69016ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-07-14T18:51:52.705 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2023-07-14T18:51:52.705 INFO:teuthology.orchestra.run.smithi078.stdout:check_obsoletes = 1 2023-07-14T18:51:52.706 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2023-07-14T18:51:52.707 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum clean all 2023-07-14T18:51:52.737 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-07-14T18:51:52.817 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-07-14T18:51:52.885 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2023-07-14T18:51:52.886 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2023-07-14T18:51:53.079 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:53.083 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:53.259 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:54.099 INFO:teuthology.orchestra.run.smithi078.stdout:44 files removed 2023-07-14T18:51:54.124 INFO:teuthology.orchestra.run.smithi088.stdout:44 files removed 2023-07-14T18:51:54.148 DEBUG:teuthology.orchestra.run.smithi078:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T18:51:54.177 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T18:51:54.344 INFO:teuthology.orchestra.run.smithi102.stdout:44 files removed 2023-07-14T18:51:54.394 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-07-14T18:51:54.479 INFO:teuthology.orchestra.run.smithi078.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:54.508 INFO:teuthology.orchestra.run.smithi088.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:54.724 INFO:teuthology.orchestra.run.smithi102.stdout:Updating Subscription Management repositories. 2023-07-14T18:51:56.247 INFO:teuthology.orchestra.run.smithi078.stdout:ceph packages for x86_64 116 kB/s | 81 kB 00:00 2023-07-14T18:51:56.364 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 120 kB/s | 81 kB 00:00 2023-07-14T18:51:56.665 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 117 kB/s | 81 kB 00:00 2023-07-14T18:51:56.823 INFO:teuthology.orchestra.run.smithi078.stdout:ceph noarch packages 17 kB/s | 9.4 kB 00:00 2023-07-14T18:51:56.966 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 17 kB/s | 9.4 kB 00:00 2023-07-14T18:51:57.166 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 20 kB/s | 9.4 kB 00:00 2023-07-14T18:51:57.224 INFO:teuthology.orchestra.run.smithi078.stdout:ceph source packages 3.9 kB/s | 1.5 kB 00:00 2023-07-14T18:51:57.366 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 3.9 kB/s | 1.5 kB 00:00 2023-07-14T18:51:57.374 INFO:teuthology.orchestra.run.smithi078.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2023-07-14T18:51:57.533 INFO:teuthology.orchestra.run.smithi088.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-07-14T18:51:57.549 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 4.1 kB/s | 1.5 kB 00:00 2023-07-14T18:51:57.709 INFO:teuthology.orchestra.run.smithi102.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-07-14T18:51:58.551 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 19 MB/s | 16 MB 00:00 2023-07-14T18:51:59.303 INFO:teuthology.orchestra.run.smithi078.stdout:Extra Packages for Enterprise Linux 8.2 MB/s | 16 MB 00:01 2023-07-14T18:52:00.329 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 5.6 MB/s | 16 MB 00:02 2023-07-14T18:52:01.951 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 387 kB/s | 24 kB 00:00 2023-07-14T18:52:02.810 INFO:teuthology.orchestra.run.smithi078.stdout:lab-extras 311 kB/s | 24 kB 00:00 2023-07-14T18:52:03.103 INFO:teuthology.orchestra.run.smithi102.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 49 MB/s | 53 MB 00:01 2023-07-14T18:52:03.800 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 386 kB/s | 24 kB 00:00 2023-07-14T18:52:04.021 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 MB/s | 47 MB 00:01 2023-07-14T18:52:05.811 INFO:teuthology.orchestra.run.smithi088.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 27 MB/s | 53 MB 00:01 2023-07-14T18:52:11.463 INFO:teuthology.orchestra.run.smithi102.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 42 MB/s | 47 MB 00:01 2023-07-14T18:52:12.197 INFO:teuthology.orchestra.run.smithi078.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 47 MB/s | 53 MB 00:01 2023-07-14T18:52:14.228 INFO:teuthology.orchestra.run.smithi088.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 43 MB/s | 47 MB 00:01 2023-07-14T18:52:22.395 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:52:22.396 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:52:22.410 INFO:teuthology.orchestra.run.smithi102.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:52:22.412 INFO:teuthology.orchestra.run.smithi102.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:52:22.413 INFO:teuthology.orchestra.run.smithi102.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:52:22.414 INFO:teuthology.orchestra.run.smithi102.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:52:22.416 INFO:teuthology.orchestra.run.smithi102.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:52:22.420 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:52:22.421 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:52:22.551 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2023-07-14T18:52:22.563 INFO:teuthology.orchestra.run.smithi102.stdout:=========================================================================================================================== 2023-07-14T18:52:22.563 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2023-07-14T18:52:22.563 INFO:teuthology.orchestra.run.smithi102.stdout:=========================================================================================================================== 2023-07-14T18:52:22.563 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2023-07-14T18:52:22.564 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:52:22.564 INFO:teuthology.orchestra.run.smithi102.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:52:22.564 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:52:22.564 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:52:22.564 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:52:22.564 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:52:22.564 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:52:22.565 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:52:22.565 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:52:22.565 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:52:22.565 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:52:22.565 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:52:22.565 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:52:22.565 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:52:22.565 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:52:22.566 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:52:22.566 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:52:22.566 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:52:22.566 INFO:teuthology.orchestra.run.smithi102.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:52:22.566 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:52:22.566 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:52:22.566 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:52:22.566 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:52:22.567 INFO:teuthology.orchestra.run.smithi102.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:52:22.567 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:52:22.567 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:52:22.567 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:52:22.567 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:52:22.567 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:52:22.567 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:52:22.568 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:52:22.568 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:52:22.568 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:52:22.568 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:52:22.568 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:52:22.568 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:52:22.568 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:52:22.568 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2023-07-14T18:52:22.569 INFO:teuthology.orchestra.run.smithi102.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:52:22.569 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:52:22.569 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:52:22.569 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2023-07-14T18:52:22.569 INFO:teuthology.orchestra.run.smithi102.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:52:22.569 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:52:22.569 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:52:22.570 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:52:22.570 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:52:22.570 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:52:22.570 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:52:22.570 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:52:22.570 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:52:22.570 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:52:22.570 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:52:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:52:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:52:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:52:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:52:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:52:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:52:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:52:22.571 INFO:teuthology.orchestra.run.smithi102.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:52:22.572 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:52:22.572 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:52:22.572 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:52:22.572 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:52:22.572 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:52:22.572 INFO:teuthology.orchestra.run.smithi102.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:52:22.572 INFO:teuthology.orchestra.run.smithi102.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:52:22.573 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:52:22.573 INFO:teuthology.orchestra.run.smithi102.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:52:22.573 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:52:22.573 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:52:22.573 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:52:22.573 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:52:22.573 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:52:22.574 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:52:22.574 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:52:22.574 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:52:22.574 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:52:22.574 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:52:22.574 INFO:teuthology.orchestra.run.smithi102.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:52:22.574 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:52:22.574 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:52:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:52:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:52:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:52:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:52:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:52:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:52:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:52:22.575 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:52:22.576 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:52:22.576 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:52:22.576 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:52:22.576 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:52:22.576 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:52:22.576 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:52:22.576 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:52:22.577 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:52:22.577 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:52:22.577 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:52:22.577 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:52:22.577 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:52:22.577 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:52:22.577 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:52:22.577 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:52:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:52:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:52:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:52:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:52:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:52:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:52:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:52:22.578 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:52:22.579 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:52:22.579 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:52:22.579 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:52:22.579 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:52:22.579 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:52:22.579 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:52:22.579 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:52:22.579 INFO:teuthology.orchestra.run.smithi102.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:52:22.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:52:22.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:52:22.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:52:22.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:52:22.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:52:22.580 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:52:22.580 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:52:22.581 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:52:22.581 INFO:teuthology.orchestra.run.smithi102.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:52:22.581 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:52:22.581 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:52:22.581 INFO:teuthology.orchestra.run.smithi102.stdout:Enabling module streams: 2023-07-14T18:52:22.581 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql 10 2023-07-14T18:52:22.581 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:52:22.581 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2023-07-14T18:52:22.582 INFO:teuthology.orchestra.run.smithi102.stdout:=========================================================================================================================== 2023-07-14T18:52:22.582 INFO:teuthology.orchestra.run.smithi102.stdout:Install 128 Packages 2023-07-14T18:52:22.582 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 3 Packages 2023-07-14T18:52:22.582 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:52:22.583 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 206 M 2023-07-14T18:52:22.583 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2023-07-14T18:52:22.864 INFO:teuthology.orchestra.run.smithi102.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 25 kB/s | 6.4 kB 00:00 2023-07-14T18:52:23.347 INFO:teuthology.orchestra.run.smithi078.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:52:23.348 INFO:teuthology.orchestra.run.smithi078.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:52:23.362 INFO:teuthology.orchestra.run.smithi078.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:52:23.363 INFO:teuthology.orchestra.run.smithi078.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:52:23.364 INFO:teuthology.orchestra.run.smithi078.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:52:23.365 INFO:teuthology.orchestra.run.smithi078.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:52:23.367 INFO:teuthology.orchestra.run.smithi078.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:52:23.370 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:52:23.371 INFO:teuthology.orchestra.run.smithi078.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:52:23.481 INFO:teuthology.orchestra.run.smithi102.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 1.3 MB/s | 850 kB 00:00 2023-07-14T18:52:23.503 INFO:teuthology.orchestra.run.smithi078.stdout:Dependencies resolved. 2023-07-14T18:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-07-14T18:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout: Package Arch Version Repository Size 2023-07-14T18:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-07-14T18:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout:Installing: 2023-07-14T18:52:23.515 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:52:23.516 INFO:teuthology.orchestra.run.smithi078.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:52:23.516 INFO:teuthology.orchestra.run.smithi078.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:52:23.516 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:52:23.516 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:52:23.516 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:52:23.516 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:52:23.516 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:52:23.517 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:52:23.517 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:52:23.517 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:52:23.517 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:52:23.517 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:52:23.517 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:52:23.517 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:52:23.517 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:52:23.518 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:52:23.518 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:52:23.518 INFO:teuthology.orchestra.run.smithi078.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:52:23.518 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:52:23.518 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:52:23.518 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:52:23.518 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:52:23.519 INFO:teuthology.orchestra.run.smithi078.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:52:23.519 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:52:23.519 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:52:23.519 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:52:23.519 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:52:23.519 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:52:23.519 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:52:23.519 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:52:23.520 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:52:23.520 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:52:23.520 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:52:23.520 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:52:23.520 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:52:23.520 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:52:23.520 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrading: 2023-07-14T18:52:23.521 INFO:teuthology.orchestra.run.smithi078.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:52:23.521 INFO:teuthology.orchestra.run.smithi078.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:52:23.521 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:52:23.521 INFO:teuthology.orchestra.run.smithi078.stdout:Installing dependencies: 2023-07-14T18:52:23.521 INFO:teuthology.orchestra.run.smithi078.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:52:23.521 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:52:23.521 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:52:23.521 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:52:23.522 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:52:23.522 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:52:23.522 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:52:23.522 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:52:23.522 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:52:23.522 INFO:teuthology.orchestra.run.smithi078.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:52:23.522 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:52:23.523 INFO:teuthology.orchestra.run.smithi078.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:52:23.523 INFO:teuthology.orchestra.run.smithi078.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:52:23.523 INFO:teuthology.orchestra.run.smithi078.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:52:23.523 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:52:23.523 INFO:teuthology.orchestra.run.smithi078.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:52:23.523 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:52:23.523 INFO:teuthology.orchestra.run.smithi078.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:52:23.523 INFO:teuthology.orchestra.run.smithi078.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:52:23.524 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:52:23.524 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:52:23.524 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:52:23.524 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:52:23.524 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:52:23.524 INFO:teuthology.orchestra.run.smithi078.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:52:23.524 INFO:teuthology.orchestra.run.smithi078.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:52:23.525 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:52:23.525 INFO:teuthology.orchestra.run.smithi078.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:52:23.525 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:52:23.525 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:52:23.525 INFO:teuthology.orchestra.run.smithi078.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:52:23.525 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:52:23.525 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:52:23.525 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:52:23.526 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:52:23.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:52:23.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:52:23.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:52:23.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:52:23.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:52:23.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:52:23.526 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:52:23.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:52:23.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:52:23.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:52:23.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:52:23.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:52:23.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:52:23.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:52:23.527 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:52:23.528 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:52:23.528 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:52:23.528 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:52:23.528 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:52:23.528 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:52:23.528 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:52:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:52:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:52:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:52:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:52:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:52:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:52:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:52:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:52:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:52:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:52:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:52:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:52:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:52:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:52:23.530 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:52:23.531 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:52:23.531 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:52:23.531 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:52:23.531 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:52:23.531 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:52:23.531 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:52:23.531 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:52:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:52:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:52:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:52:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:52:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:52:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:52:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:52:23.532 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:52:23.533 INFO:teuthology.orchestra.run.smithi078.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:52:23.533 INFO:teuthology.orchestra.run.smithi078.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:52:23.533 INFO:teuthology.orchestra.run.smithi078.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:52:23.533 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:52:23.533 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:52:23.533 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling module streams: 2023-07-14T18:52:23.533 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql 10 2023-07-14T18:52:23.533 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:52:23.534 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction Summary 2023-07-14T18:52:23.534 INFO:teuthology.orchestra.run.smithi078.stdout:=========================================================================================================================== 2023-07-14T18:52:23.534 INFO:teuthology.orchestra.run.smithi078.stdout:Install 128 Packages 2023-07-14T18:52:23.534 INFO:teuthology.orchestra.run.smithi078.stdout:Upgrade 3 Packages 2023-07-14T18:52:23.534 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:52:23.535 INFO:teuthology.orchestra.run.smithi078.stdout:Total download size: 206 M 2023-07-14T18:52:23.535 INFO:teuthology.orchestra.run.smithi078.stdout:Downloading Packages: 2023-07-14T18:52:23.598 INFO:teuthology.orchestra.run.smithi102.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 1.4 MB/s | 167 kB 00:00 2023-07-14T18:52:23.806 INFO:teuthology.orchestra.run.smithi078.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 27 kB/s | 6.4 kB 00:00 2023-07-14T18:52:24.000 INFO:teuthology.orchestra.run.smithi102.stdout:(4/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 3.9 MB/s | 5.4 MB 00:01 2023-07-14T18:52:24.248 INFO:teuthology.orchestra.run.smithi078.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 1.9 MB/s | 850 kB 00:00 2023-07-14T18:52:24.265 INFO:teuthology.orchestra.run.smithi102.stdout:(5/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 5.9 MB/s | 1.6 MB 00:00 2023-07-14T18:52:24.332 INFO:teuthology.orchestra.run.smithi078.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 2.0 MB/s | 167 kB 00:00 2023-07-14T18:52:24.507 INFO:teuthology.orchestra.run.smithi102.stdout:(6/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 2.5 MB/s | 2.3 MB 00:00 2023-07-14T18:52:25.032 INFO:teuthology.orchestra.run.smithi102.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 6.0 MB/s | 4.6 MB 00:00 2023-07-14T18:52:25.109 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:52:25.110 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-07-14T18:52:25.125 INFO:teuthology.orchestra.run.smithi088.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-07-14T18:52:25.126 INFO:teuthology.orchestra.run.smithi088.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-07-14T18:52:25.127 INFO:teuthology.orchestra.run.smithi088.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-07-14T18:52:25.129 INFO:teuthology.orchestra.run.smithi088.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-07-14T18:52:25.131 INFO:teuthology.orchestra.run.smithi088.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-07-14T18:52:25.134 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-07-14T18:52:25.135 INFO:teuthology.orchestra.run.smithi088.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-07-14T18:52:25.257 INFO:teuthology.orchestra.run.smithi078.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 2.5 MB/s | 2.3 MB 00:00 2023-07-14T18:52:25.269 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2023-07-14T18:52:25.281 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-14T18:52:25.282 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2023-07-14T18:52:25.282 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-14T18:52:25.282 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2023-07-14T18:52:25.282 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-07-14T18:52:25.282 INFO:teuthology.orchestra.run.smithi088.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-07-14T18:52:25.282 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.4 k 2023-07-14T18:52:25.282 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 5.4 M 2023-07-14T18:52:25.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 850 k 2023-07-14T18:52:25.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 167 k 2023-07-14T18:52:25.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 1.6 M 2023-07-14T18:52:25.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 129 k 2023-07-14T18:52:25.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 1.7 M 2023-07-14T18:52:25.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 7.4 M 2023-07-14T18:52:25.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 47 k 2023-07-14T18:52:25.283 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 14 M 2023-07-14T18:52:25.284 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 48 M 2023-07-14T18:52:25.284 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 258 k 2023-07-14T18:52:25.284 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 165 k 2023-07-14T18:52:25.284 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 260 k 2023-07-14T18:52:25.284 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 18 k 2023-07-14T18:52:25.284 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-07-14T18:52:25.284 INFO:teuthology.orchestra.run.smithi088.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-07-14T18:52:25.285 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-07-14T18:52:25.285 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 31 k 2023-07-14T18:52:25.285 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 772 k 2023-07-14T18:52:25.285 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 139 k 2023-07-14T18:52:25.285 INFO:teuthology.orchestra.run.smithi088.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-07-14T18:52:25.285 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-07-14T18:52:25.285 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-07-14T18:52:25.285 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-07-14T18:52:25.286 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-07-14T18:52:25.286 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-07-14T18:52:25.286 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 197 k 2023-07-14T18:52:25.286 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 371 k 2023-07-14T18:52:25.286 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 364 k 2023-07-14T18:52:25.286 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 107 k 2023-07-14T18:52:25.286 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 94 k 2023-07-14T18:52:25.286 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.5 M 2023-07-14T18:52:25.287 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 179 k 2023-07-14T18:52:25.287 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-07-14T18:52:25.287 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2023-07-14T18:52:25.287 INFO:teuthology.orchestra.run.smithi088.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-07-14T18:52:25.287 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.7 M 2023-07-14T18:52:25.287 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 3.6 M 2023-07-14T18:52:25.287 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2023-07-14T18:52:25.287 INFO:teuthology.orchestra.run.smithi088.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-07-14T18:52:25.288 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 22 M 2023-07-14T18:52:25.288 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 23 k 2023-07-14T18:52:25.288 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 2.3 M 2023-07-14T18:52:25.288 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 241 k 2023-07-14T18:52:25.288 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 4.6 M 2023-07-14T18:52:25.288 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 18 M 2023-07-14T18:52:25.288 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:18.0.0-4940.g81c9387d.el8 ceph-noarch 15 k 2023-07-14T18:52:25.289 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 24 k 2023-07-14T18:52:25.289 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-07-14T18:52:25.289 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-07-14T18:52:25.289 INFO:teuthology.orchestra.run.smithi088.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-07-14T18:52:25.289 INFO:teuthology.orchestra.run.smithi088.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-07-14T18:52:25.289 INFO:teuthology.orchestra.run.smithi088.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-07-14T18:52:25.289 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 135 k 2023-07-14T18:52:25.289 INFO:teuthology.orchestra.run.smithi088.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-07-14T18:52:25.290 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-07-14T18:52:25.290 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-07-14T18:52:25.290 INFO:teuthology.orchestra.run.smithi088.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-07-14T18:52:25.290 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-07-14T18:52:25.290 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-07-14T18:52:25.290 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 473 k 2023-07-14T18:52:25.290 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-07-14T18:52:25.291 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 6.8 M 2023-07-14T18:52:25.291 INFO:teuthology.orchestra.run.smithi088.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-07-14T18:52:25.291 INFO:teuthology.orchestra.run.smithi088.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-07-14T18:52:25.291 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-07-14T18:52:25.291 INFO:teuthology.orchestra.run.smithi088.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-07-14T18:52:25.291 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-07-14T18:52:25.291 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-07-14T18:52:25.291 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-07-14T18:52:25.292 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-07-14T18:52:25.292 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-07-14T18:52:25.292 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-07-14T18:52:25.292 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-07-14T18:52:25.292 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-07-14T18:52:25.292 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-07-14T18:52:25.292 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-07-14T18:52:25.292 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-07-14T18:52:25.293 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-07-14T18:52:25.293 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 45 k 2023-07-14T18:52:25.293 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:18.0.0-4940.g81c9387d.el8 ceph 122 k 2023-07-14T18:52:25.293 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-07-14T18:52:25.293 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-07-14T18:52:25.293 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-07-14T18:52:25.293 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-07-14T18:52:25.294 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-07-14T18:52:25.294 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-07-14T18:52:25.294 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-07-14T18:52:25.294 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-07-14T18:52:25.294 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-07-14T18:52:25.294 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-07-14T18:52:25.294 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-07-14T18:52:25.294 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:52:25.295 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-07-14T18:52:25.295 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-07-14T18:52:25.295 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-07-14T18:52:25.295 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-07-14T18:52:25.295 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-07-14T18:52:25.295 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-07-14T18:52:25.295 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-07-14T18:52:25.295 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-07-14T18:52:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-07-14T18:52:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-07-14T18:52:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-07-14T18:52:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-07-14T18:52:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-07-14T18:52:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-07-14T18:52:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-07-14T18:52:25.296 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-07-14T18:52:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-07-14T18:52:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-07-14T18:52:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-07-14T18:52:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-07-14T18:52:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-07-14T18:52:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-07-14T18:52:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-07-14T18:52:25.297 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-07-14T18:52:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-07-14T18:52:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-07-14T18:52:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-07-14T18:52:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-07-14T18:52:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-07-14T18:52:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-07-14T18:52:25.298 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-07-14T18:52:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-07-14T18:52:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-07-14T18:52:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-07-14T18:52:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-07-14T18:52:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-07-14T18:52:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-07-14T18:52:25.299 INFO:teuthology.orchestra.run.smithi088.stdout:Enabling module streams: 2023-07-14T18:52:25.299 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql 10 2023-07-14T18:52:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:52:25.300 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2023-07-14T18:52:25.300 INFO:teuthology.orchestra.run.smithi088.stdout:=========================================================================================================================== 2023-07-14T18:52:25.300 INFO:teuthology.orchestra.run.smithi088.stdout:Install 128 Packages 2023-07-14T18:52:25.300 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 3 Packages 2023-07-14T18:52:25.300 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:52:25.301 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 206 M 2023-07-14T18:52:25.301 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2023-07-14T18:52:25.599 INFO:teuthology.orchestra.run.smithi078.stdout:(5/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 2.7 MB/s | 5.4 MB 00:02 2023-07-14T18:52:25.791 INFO:teuthology.orchestra.run.smithi078.stdout:(6/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 2.9 MB/s | 1.6 MB 00:00 2023-07-14T18:52:26.103 INFO:teuthology.orchestra.run.smithi088.stdout:(1/131): ceph-18.0.0-4940.g81c9387d.el8.x86_64. 16 kB/s | 6.4 kB 00:00 2023-07-14T18:52:26.149 INFO:teuthology.orchestra.run.smithi102.stdout:(8/131): ceph-common-18.0.0-4940.g81c9387d.el8. 6.3 MB/s | 22 MB 00:03 2023-07-14T18:52:26.249 INFO:teuthology.orchestra.run.smithi102.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 243 kB/s | 24 kB 00:00 2023-07-14T18:52:27.047 INFO:teuthology.orchestra.run.smithi078.stdout:(7/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 3.2 MB/s | 4.6 MB 00:01 2023-07-14T18:52:27.153 INFO:teuthology.orchestra.run.smithi088.stdout:(2/131): ceph-fuse-18.0.0-4940.g81c9387d.el8.x8 810 kB/s | 850 kB 00:01 2023-07-14T18:52:27.378 INFO:teuthology.orchestra.run.smithi088.stdout:(3/131): ceph-immutable-object-cache-18.0.0-494 743 kB/s | 167 kB 00:00 2023-07-14T18:52:28.100 INFO:teuthology.orchestra.run.smithi102.stdout:(10/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 4.4 MB/s | 14 MB 00:03 2023-07-14T18:52:28.450 INFO:teuthology.orchestra.run.smithi102.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 743 kB/s | 260 kB 00:00 2023-07-14T18:52:28.600 INFO:teuthology.orchestra.run.smithi102.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 207 kB/s | 31 kB 00:00 2023-07-14T18:52:28.959 INFO:teuthology.orchestra.run.smithi102.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 2.1 MB/s | 772 kB 00:00 2023-07-14T18:52:29.034 INFO:teuthology.orchestra.run.smithi102.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 1.8 MB/s | 135 kB 00:00 2023-07-14T18:52:29.135 INFO:teuthology.orchestra.run.smithi102.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 1.4 MB/s | 139 kB 00:00 2023-07-14T18:52:29.418 INFO:teuthology.orchestra.run.smithi102.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 1.6 MB/s | 473 kB 00:00 2023-07-14T18:52:29.628 INFO:teuthology.orchestra.run.smithi088.stdout:(4/131): ceph-mds-18.0.0-4940.g81c9387d.el8.x86 1.0 MB/s | 2.3 MB 00:02 2023-07-14T18:52:29.970 INFO:teuthology.orchestra.run.smithi088.stdout:(5/131): ceph-base-18.0.0-4940.g81c9387d.el8.x8 1.3 MB/s | 5.4 MB 00:04 2023-07-14T18:52:31.003 INFO:teuthology.orchestra.run.smithi088.stdout:(6/131): ceph-mgr-18.0.0-4940.g81c9387d.el8.x86 1.1 MB/s | 1.6 MB 00:01 2023-07-14T18:52:31.470 INFO:teuthology.orchestra.run.smithi088.stdout:(7/131): ceph-common-18.0.0-4940.g81c9387d.el8. 3.8 MB/s | 22 MB 00:05 2023-07-14T18:52:32.098 INFO:teuthology.orchestra.run.smithi078.stdout:(8/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x86 2.8 MB/s | 18 MB 00:06 2023-07-14T18:52:32.201 INFO:teuthology.orchestra.run.smithi102.stdout:(17/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 2.4 MB/s | 6.8 MB 00:02 2023-07-14T18:52:32.208 INFO:teuthology.orchestra.run.smithi078.stdout:(9/131): ceph-selinux-18.0.0-4940.g81c9387d.el8 223 kB/s | 24 kB 00:00 2023-07-14T18:52:32.302 INFO:teuthology.orchestra.run.smithi102.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 448 kB/s | 45 kB 00:00 2023-07-14T18:52:32.410 INFO:teuthology.orchestra.run.smithi102.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 1.1 MB/s | 122 kB 00:00 2023-07-14T18:52:32.491 INFO:teuthology.orchestra.run.smithi078.stdout:(10/131): ceph-common-18.0.0-4940.g81c9387d.el8 2.5 MB/s | 22 MB 00:08 2023-07-14T18:52:32.510 INFO:teuthology.orchestra.run.smithi102.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 1.9 MB/s | 197 kB 00:00 2023-07-14T18:52:32.703 INFO:teuthology.orchestra.run.smithi102.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 1.9 MB/s | 371 kB 00:00 2023-07-14T18:52:32.758 INFO:teuthology.orchestra.run.smithi078.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 977 kB/s | 260 kB 00:00 2023-07-14T18:52:32.867 INFO:teuthology.orchestra.run.smithi078.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 287 kB/s | 31 kB 00:00 2023-07-14T18:52:32.894 INFO:teuthology.orchestra.run.smithi102.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 1.9 MB/s | 364 kB 00:00 2023-07-14T18:52:33.003 INFO:teuthology.orchestra.run.smithi102.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 984 kB/s | 107 kB 00:00 2023-07-14T18:52:33.112 INFO:teuthology.orchestra.run.smithi102.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 864 kB/s | 94 kB 00:00 2023-07-14T18:52:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 1.6 MB/s | 772 kB 00:00 2023-07-14T18:52:33.492 INFO:teuthology.orchestra.run.smithi078.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 903 kB/s | 135 kB 00:00 2023-07-14T18:52:33.609 INFO:teuthology.orchestra.run.smithi078.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 1.2 MB/s | 139 kB 00:00 2023-07-14T18:52:33.645 INFO:teuthology.orchestra.run.smithi088.stdout:(8/131): ceph-mon-18.0.0-4940.g81c9387d.el8.x86 1.3 MB/s | 4.6 MB 00:03 2023-07-14T18:52:33.787 INFO:teuthology.orchestra.run.smithi088.stdout:(9/131): ceph-radosgw-18.0.0-4940.g81c9387d.el8 5.8 MB/s | 14 MB 00:02 2023-07-14T18:52:33.814 INFO:teuthology.orchestra.run.smithi088.stdout:(10/131): ceph-selinux-18.0.0-4940.g81c9387d.el 146 kB/s | 24 kB 00:00 2023-07-14T18:52:33.851 INFO:teuthology.orchestra.run.smithi078.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 1.9 MB/s | 473 kB 00:00 2023-07-14T18:52:34.104 INFO:teuthology.orchestra.run.smithi088.stdout:(11/131): cephfs-mirror-18.0.0-4940.g81c9387d.e 892 kB/s | 260 kB 00:00 2023-07-14T18:52:34.221 INFO:teuthology.orchestra.run.smithi088.stdout:(12/131): libcephfs-devel-18.0.0-4940.g81c9387d 266 kB/s | 31 kB 00:00 2023-07-14T18:52:34.787 INFO:teuthology.orchestra.run.smithi102.stdout:(25/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 2.1 MB/s | 3.5 MB 00:01 2023-07-14T18:52:34.904 INFO:teuthology.orchestra.run.smithi102.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.5 MB/s | 179 kB 00:00 2023-07-14T18:52:34.913 INFO:teuthology.orchestra.run.smithi088.stdout:(13/131): libcephfs2-18.0.0-4940.g81c9387d.el8. 1.1 MB/s | 772 kB 00:00 2023-07-14T18:52:35.004 INFO:teuthology.orchestra.run.smithi102.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 230 kB/s | 23 kB 00:00 2023-07-14T18:52:35.060 INFO:teuthology.orchestra.run.smithi078.stdout:(17/131): ceph-radosgw-18.0.0-4940.g81c9387d.el 1.7 MB/s | 14 MB 00:08 2023-07-14T18:52:35.113 INFO:teuthology.orchestra.run.smithi102.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 1.2 MB/s | 129 kB 00:00 2023-07-14T18:52:35.114 INFO:teuthology.orchestra.run.smithi088.stdout:(14/131): libcephsqlite-18.0.0-4940.g81c9387d.e 678 kB/s | 135 kB 00:00 2023-07-14T18:52:35.177 INFO:teuthology.orchestra.run.smithi078.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 384 kB/s | 45 kB 00:00 2023-07-14T18:52:35.311 INFO:teuthology.orchestra.run.smithi078.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 920 kB/s | 122 kB 00:00 2023-07-14T18:52:35.330 INFO:teuthology.orchestra.run.smithi088.stdout:(15/131): librados-devel-18.0.0-4940.g81c9387d. 642 kB/s | 139 kB 00:00 2023-07-14T18:52:35.528 INFO:teuthology.orchestra.run.smithi078.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 909 kB/s | 197 kB 00:00 2023-07-14T18:52:35.747 INFO:teuthology.orchestra.run.smithi088.stdout:(16/131): libradosstriper1-18.0.0-4940.g81c9387 1.1 MB/s | 473 kB 00:00 2023-07-14T18:52:35.838 INFO:teuthology.orchestra.run.smithi078.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 1.2 MB/s | 371 kB 00:00 2023-07-14T18:52:36.013 INFO:teuthology.orchestra.run.smithi102.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.9 MB/s | 1.7 MB 00:00 2023-07-14T18:52:36.128 INFO:teuthology.orchestra.run.smithi078.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 1.2 MB/s | 364 kB 00:00 2023-07-14T18:52:36.245 INFO:teuthology.orchestra.run.smithi078.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 915 kB/s | 107 kB 00:00 2023-07-14T18:52:36.371 INFO:teuthology.orchestra.run.smithi078.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 749 kB/s | 94 kB 00:00 2023-07-14T18:52:37.405 INFO:teuthology.orchestra.run.smithi102.stdout:(30/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 1.4 MB/s | 18 MB 00:12 2023-07-14T18:52:37.739 INFO:teuthology.orchestra.run.smithi102.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 723 kB/s | 241 kB 00:00 2023-07-14T18:52:37.848 INFO:teuthology.orchestra.run.smithi102.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 438 kB/s | 47 kB 00:00 2023-07-14T18:52:37.854 INFO:teuthology.orchestra.run.smithi078.stdout:(25/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 1.7 MB/s | 6.8 MB 00:04 2023-07-14T18:52:37.964 INFO:teuthology.orchestra.run.smithi102.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 125 kB/s | 15 kB 00:00 2023-07-14T18:52:38.071 INFO:teuthology.orchestra.run.smithi078.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 825 kB/s | 179 kB 00:00 2023-07-14T18:52:38.196 INFO:teuthology.orchestra.run.smithi078.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 184 kB/s | 23 kB 00:00 2023-07-14T18:52:38.264 INFO:teuthology.orchestra.run.smithi102.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 861 kB/s | 258 kB 00:00 2023-07-14T18:52:38.305 INFO:teuthology.orchestra.run.smithi078.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 1.2 MB/s | 129 kB 00:00 2023-07-14T18:52:38.465 INFO:teuthology.orchestra.run.smithi102.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 826 kB/s | 165 kB 00:00 2023-07-14T18:52:38.557 INFO:teuthology.orchestra.run.smithi102.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 196 kB/s | 18 kB 00:00 2023-07-14T18:52:38.864 INFO:teuthology.orchestra.run.smithi078.stdout:(29/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 1.4 MB/s | 3.5 MB 00:02 2023-07-14T18:52:39.182 INFO:teuthology.orchestra.run.smithi102.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 490 kB/s | 306 kB 00:00 2023-07-14T18:52:39.266 INFO:teuthology.orchestra.run.smithi102.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 706 kB/s | 59 kB 00:00 2023-07-14T18:52:39.341 INFO:teuthology.orchestra.run.smithi102.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.0 MB/s | 75 kB 00:00 2023-07-14T18:52:39.389 INFO:teuthology.orchestra.run.smithi078.stdout:(30/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 1.6 MB/s | 1.7 MB 00:01 2023-07-14T18:52:39.589 INFO:teuthology.orchestra.run.smithi078.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 1.2 MB/s | 241 kB 00:00 2023-07-14T18:52:39.666 INFO:teuthology.orchestra.run.smithi102.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 1.4 MB/s | 476 kB 00:00 2023-07-14T18:52:39.706 INFO:teuthology.orchestra.run.smithi078.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 406 kB/s | 47 kB 00:00 2023-07-14T18:52:39.742 INFO:teuthology.orchestra.run.smithi102.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 582 kB/s | 44 kB 00:00 2023-07-14T18:52:39.867 INFO:teuthology.orchestra.run.smithi102.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2023-07-14T18:52:39.942 INFO:teuthology.orchestra.run.smithi102.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 440 kB/s | 33 kB 00:00 2023-07-14T18:52:40.048 INFO:teuthology.orchestra.run.smithi078.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 42 kB/s | 15 kB 00:00 2023-07-14T18:52:40.051 INFO:teuthology.orchestra.run.smithi102.stdout:(44/131): ceph-mgr-diskprediction-local-18.0.0- 1.8 MB/s | 7.4 MB 00:04 2023-07-14T18:52:40.076 INFO:teuthology.orchestra.run.smithi102.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 122 kB/s | 16 kB 00:00 2023-07-14T18:52:40.302 INFO:teuthology.orchestra.run.smithi102.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.7 MB/s | 384 kB 00:00 2023-07-14T18:52:40.332 INFO:teuthology.orchestra.run.smithi078.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 912 kB/s | 258 kB 00:00 2023-07-14T18:52:40.402 INFO:teuthology.orchestra.run.smithi102.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 814 kB/s | 81 kB 00:00 2023-07-14T18:52:40.427 INFO:teuthology.orchestra.run.smithi102.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 459 kB/s | 173 kB 00:00 2023-07-14T18:52:40.457 INFO:teuthology.orchestra.run.smithi078.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 1.3 MB/s | 165 kB 00:00 2023-07-14T18:52:40.477 INFO:teuthology.orchestra.run.smithi102.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2023-07-14T18:52:40.503 INFO:teuthology.orchestra.run.smithi102.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 234 kB/s | 18 kB 00:00 2023-07-14T18:52:40.566 INFO:teuthology.orchestra.run.smithi078.stdout:(36/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 166 kB/s | 18 kB 00:00 2023-07-14T18:52:40.578 INFO:teuthology.orchestra.run.smithi102.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 650 kB/s | 49 kB 00:00 2023-07-14T18:52:40.653 INFO:teuthology.orchestra.run.smithi102.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 783 kB/s | 59 kB 00:00 2023-07-14T18:52:40.729 INFO:teuthology.orchestra.run.smithi102.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 803 kB/s | 60 kB 00:00 2023-07-14T18:52:40.808 INFO:teuthology.orchestra.run.smithi078.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2023-07-14T18:52:40.867 INFO:teuthology.orchestra.run.smithi078.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.0 MB/s | 59 kB 00:00 2023-07-14T18:52:40.930 INFO:teuthology.orchestra.run.smithi088.stdout:(17/131): librgw2-18.0.0-4940.g81c9387d.el8.x86 1.3 MB/s | 6.8 MB 00:05 2023-07-14T18:52:40.934 INFO:teuthology.orchestra.run.smithi078.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-07-14T18:52:40.954 INFO:teuthology.orchestra.run.smithi102.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-07-14T18:52:41.021 INFO:teuthology.orchestra.run.smithi102.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 234 kB/s | 16 kB 00:00 2023-07-14T18:52:41.034 INFO:teuthology.orchestra.run.smithi078.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.7 MB/s | 476 kB 00:00 2023-07-14T18:52:41.055 INFO:teuthology.orchestra.run.smithi088.stdout:(18/131): python3-ceph-argparse-18.0.0-4940.g81 358 kB/s | 45 kB 00:00 2023-07-14T18:52:41.063 INFO:teuthology.orchestra.run.smithi102.stdout:(56/131): python3-kubernetes-11.0.0-6.el8.noarc 2.0 MB/s | 1.2 MB 00:00 2023-07-14T18:52:41.088 INFO:teuthology.orchestra.run.smithi102.stdout:(57/131): python3-repoze-lru-0.7-6.el8.noarch.r 502 kB/s | 34 kB 00:00 2023-07-14T18:52:41.092 INFO:teuthology.orchestra.run.smithi078.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 749 kB/s | 44 kB 00:00 2023-07-14T18:52:41.151 INFO:teuthology.orchestra.run.smithi078.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2023-07-14T18:52:41.155 INFO:teuthology.orchestra.run.smithi102.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 923 kB/s | 61 kB 00:00 2023-07-14T18:52:41.189 INFO:teuthology.orchestra.run.smithi102.stdout:(59/131): python3-routes-2.4.1-12.el8.noarch.rp 1.5 MB/s | 196 kB 00:00 2023-07-14T18:52:41.210 INFO:teuthology.orchestra.run.smithi078.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 566 kB/s | 33 kB 00:00 2023-07-14T18:52:41.222 INFO:teuthology.orchestra.run.smithi102.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 280 kB/s | 19 kB 00:00 2023-07-14T18:52:41.256 INFO:teuthology.orchestra.run.smithi102.stdout:(61/131): python3-singledispatch-3.4.0.3-18.el8 358 kB/s | 24 kB 00:00 2023-07-14T18:52:41.263 INFO:teuthology.orchestra.run.smithi088.stdout:(19/131): python3-ceph-common-18.0.0-4940.g81c9 589 kB/s | 122 kB 00:00 2023-07-14T18:52:41.268 INFO:teuthology.orchestra.run.smithi078.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 279 kB/s | 16 kB 00:00 2023-07-14T18:52:41.289 INFO:teuthology.orchestra.run.smithi102.stdout:(62/131): python3-tempora-1.14.1-5.el8.noarch.r 438 kB/s | 29 kB 00:00 2023-07-14T18:52:41.323 INFO:teuthology.orchestra.run.smithi102.stdout:(63/131): python3-trustme-0.6.0-4.el8.noarch.rp 411 kB/s | 27 kB 00:00 2023-07-14T18:52:41.327 INFO:teuthology.orchestra.run.smithi078.stdout:(45/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2023-07-14T18:52:41.402 INFO:teuthology.orchestra.run.smithi078.stdout:(46/131): python3-cherrypy-18.4.0-1.el8.noarch. 5.0 MB/s | 384 kB 00:00 2023-07-14T18:52:41.461 INFO:teuthology.orchestra.run.smithi078.stdout:(47/131): python3-google-auth-1.1.1-10.el8.noar 1.4 MB/s | 81 kB 00:00 2023-07-14T18:52:41.486 INFO:teuthology.orchestra.run.smithi102.stdout:(64/131): python3-waitress-1.4.3-1.el8.noarch.r 1.3 MB/s | 255 kB 00:00 2023-07-14T18:52:41.489 INFO:teuthology.orchestra.run.smithi088.stdout:(20/131): python3-cephfs-18.0.0-4940.g81c9387d. 874 kB/s | 197 kB 00:00 2023-07-14T18:52:41.503 INFO:teuthology.orchestra.run.smithi078.stdout:(48/131): python3-jaraco-6.2-6.el8.noarch.rpm 267 kB/s | 11 kB 00:00 2023-07-14T18:52:41.515 INFO:teuthology.orchestra.run.smithi102.stdout:(65/131): python3-webob-1.8.5-1.el8.1.noarch.rp 1.3 MB/s | 251 kB 00:00 2023-07-14T18:52:41.553 INFO:teuthology.orchestra.run.smithi078.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 351 kB/s | 18 kB 00:00 2023-07-14T18:52:41.558 INFO:teuthology.orchestra.run.smithi102.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 861 kB/s | 61 kB 00:00 2023-07-14T18:52:41.591 INFO:teuthology.orchestra.run.smithi102.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2023-07-14T18:52:41.625 INFO:teuthology.orchestra.run.smithi102.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 343 kB/s | 23 kB 00:00 2023-07-14T18:52:41.650 INFO:teuthology.orchestra.run.smithi102.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T18:52:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:(50/131): python3-kubernetes-11.0.0-6.el8.noarc 10 MB/s | 1.2 MB 00:00 2023-07-14T18:52:41.729 INFO:teuthology.orchestra.run.smithi078.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 832 kB/s | 49 kB 00:00 2023-07-14T18:52:41.788 INFO:teuthology.orchestra.run.smithi078.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-07-14T18:52:41.806 INFO:teuthology.orchestra.run.smithi088.stdout:(21/131): python3-rados-18.0.0-4940.g81c9387d.e 1.1 MB/s | 371 kB 00:00 2023-07-14T18:52:41.809 INFO:teuthology.orchestra.run.smithi102.stdout:(70/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 296 kB/s | 47 kB 00:00 2023-07-14T18:52:41.855 INFO:teuthology.orchestra.run.smithi078.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 905 kB/s | 60 kB 00:00 2023-07-14T18:52:41.913 INFO:teuthology.orchestra.run.smithi078.stdout:(54/131): python3-pecan-1.3.2-9.el8.noarch.rpm 4.7 MB/s | 283 kB 00:00 2023-07-14T18:52:41.967 INFO:teuthology.orchestra.run.smithi102.stdout:(71/131): libselinux-devel-2.9-5.el8.x86_64.rpm 1.2 MB/s | 200 kB 00:00 2023-07-14T18:52:41.980 INFO:teuthology.orchestra.run.smithi078.stdout:(55/131): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2023-07-14T18:52:42.039 INFO:teuthology.orchestra.run.smithi078.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 578 kB/s | 34 kB 00:00 2023-07-14T18:52:42.098 INFO:teuthology.orchestra.run.smithi078.stdout:(57/131): python3-routes-2.4.1-12.el8.noarch.rp 3.3 MB/s | 196 kB 00:00 2023-07-14T18:52:42.101 INFO:teuthology.orchestra.run.smithi102.stdout:(72/131): python3-jwt-1.6.1-2.el8.noarch.rpm 320 kB/s | 43 kB 00:00 2023-07-14T18:52:42.156 INFO:teuthology.orchestra.run.smithi078.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2023-07-14T18:52:42.215 INFO:teuthology.orchestra.run.smithi078.stdout:(59/131): python3-simplegeneric-0.8.1-17.el8.no 321 kB/s | 19 kB 00:00 2023-07-14T18:52:42.239 INFO:teuthology.orchestra.run.smithi088.stdout:(22/131): python3-rbd-18.0.0-4940.g81c9387d.el8 841 kB/s | 364 kB 00:00 2023-07-14T18:52:42.243 INFO:teuthology.orchestra.run.smithi102.stdout:(73/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2023-07-14T18:52:42.309 INFO:teuthology.orchestra.run.smithi078.stdout:(60/131): python3-singledispatch-3.4.0.3-18.el8 257 kB/s | 24 kB 00:00 2023-07-14T18:52:42.365 INFO:teuthology.orchestra.run.smithi088.stdout:(23/131): python3-rgw-18.0.0-4940.g81c9387d.el8 854 kB/s | 107 kB 00:00 2023-07-14T18:52:42.366 INFO:teuthology.orchestra.run.smithi078.stdout:(61/131): python3-tempora-1.14.1-5.el8.noarch.r 514 kB/s | 29 kB 00:00 2023-07-14T18:52:42.393 INFO:teuthology.orchestra.run.smithi102.stdout:(74/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 321 kB/s | 48 kB 00:00 2023-07-14T18:52:42.424 INFO:teuthology.orchestra.run.smithi078.stdout:(62/131): python3-trustme-0.6.0-4.el8.noarch.rp 474 kB/s | 27 kB 00:00 2023-07-14T18:52:42.481 INFO:teuthology.orchestra.run.smithi088.stdout:(24/131): rbd-fuse-18.0.0-4940.g81c9387d.el8.x8 803 kB/s | 94 kB 00:00 2023-07-14T18:52:42.483 INFO:teuthology.orchestra.run.smithi078.stdout:(63/131): python3-waitress-1.4.3-1.el8.noarch.r 4.3 MB/s | 255 kB 00:00 2023-07-14T18:52:42.527 INFO:teuthology.orchestra.run.smithi102.stdout:(75/131): thrift-0.13.0-2.el8.x86_64.rpm 1.9 MB/s | 1.7 MB 00:00 2023-07-14T18:52:42.541 INFO:teuthology.orchestra.run.smithi078.stdout:(64/131): python3-webob-1.8.5-1.el8.1.noarch.rp 4.2 MB/s | 251 kB 00:00 2023-07-14T18:52:42.552 INFO:teuthology.orchestra.run.smithi102.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 548 kB/s | 87 kB 00:00 2023-07-14T18:52:42.600 INFO:teuthology.orchestra.run.smithi078.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2023-07-14T18:52:42.659 INFO:teuthology.orchestra.run.smithi078.stdout:(66/131): python3-webtest-2.0.33-1.el8.noarch.r 1.4 MB/s | 85 kB 00:00 2023-07-14T18:52:42.686 INFO:teuthology.orchestra.run.smithi102.stdout:(77/131): python3-pycparser-2.14-14.el8.noarch. 854 kB/s | 109 kB 00:00 2023-07-14T18:52:42.718 INFO:teuthology.orchestra.run.smithi078.stdout:(67/131): python3-zc-lockfile-2.0-2.el8.noarch. 393 kB/s | 23 kB 00:00 2023-07-14T18:52:42.736 INFO:teuthology.orchestra.run.smithi102.stdout:(78/131): python3-requests-oauthlib-1.0.0-1.el8 204 kB/s | 43 kB 00:00 2023-07-14T18:52:42.835 INFO:teuthology.orchestra.run.smithi078.stdout:(68/131): thrift-0.13.0-2.el8.x86_64.rpm 15 MB/s | 1.7 MB 00:00 2023-07-14T18:52:42.845 INFO:teuthology.orchestra.run.smithi102.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 1.4 MB/s | 229 kB 00:00 2023-07-14T18:52:42.860 INFO:teuthology.orchestra.run.smithi078.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T18:52:42.879 INFO:teuthology.orchestra.run.smithi102.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 3.9 MB/s | 560 kB 00:00 2023-07-14T18:52:42.912 INFO:teuthology.orchestra.run.smithi102.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 279 kB/s | 19 kB 00:00 2023-07-14T18:52:42.987 INFO:teuthology.orchestra.run.smithi102.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 1.4 MB/s | 155 kB 00:00 2023-07-14T18:52:43.021 INFO:teuthology.orchestra.run.smithi102.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 5.0 MB/s | 559 kB 00:00 2023-07-14T18:52:43.130 INFO:teuthology.orchestra.run.smithi102.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 126 kB/s | 18 kB 00:00 2023-07-14T18:52:43.164 INFO:teuthology.orchestra.run.smithi102.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.2 MB/s | 605 kB 00:00 2023-07-14T18:52:43.189 INFO:teuthology.orchestra.run.smithi102.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 4.0 MB/s | 238 kB 00:00 2023-07-14T18:52:43.223 INFO:teuthology.orchestra.run.smithi088.stdout:(25/131): ceph-test-18.0.0-4940.g81c9387d.el8.x 5.1 MB/s | 48 MB 00:09 2023-07-14T18:52:43.239 INFO:teuthology.orchestra.run.smithi102.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.4 MB/s | 72 kB 00:00 2023-07-14T18:52:43.289 INFO:teuthology.orchestra.run.smithi102.stdout:(88/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 19 MB/s | 2.3 MB 00:00 2023-07-14T18:52:43.315 INFO:teuthology.orchestra.run.smithi102.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-07-14T18:52:43.318 INFO:teuthology.orchestra.run.smithi078.stdout:(70/131): python3-scipy-1.0.0-21.module+el8.5.0 30 MB/s | 14 MB 00:00 2023-07-14T18:52:43.340 INFO:teuthology.orchestra.run.smithi102.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-07-14T18:52:43.357 INFO:teuthology.orchestra.run.smithi088.stdout:(26/131): rbd-nbd-18.0.0-4940.g81c9387d.el8.x86 1.3 MB/s | 179 kB 00:00 2023-07-14T18:52:43.390 INFO:teuthology.orchestra.run.smithi102.stdout:(91/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-07-14T18:52:43.402 INFO:teuthology.orchestra.run.smithi078.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.3 MB/s | 110 kB 00:00 2023-07-14T18:52:43.416 INFO:teuthology.orchestra.run.smithi102.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 974 kB/s | 98 kB 00:00 2023-07-14T18:52:43.441 INFO:teuthology.orchestra.run.smithi102.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 764 kB/s | 39 kB 00:00 2023-07-14T18:52:43.469 INFO:teuthology.orchestra.run.smithi078.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 813 kB/s | 54 kB 00:00 2023-07-14T18:52:43.474 INFO:teuthology.orchestra.run.smithi088.stdout:(27/131): ceph-grafana-dashboards-18.0.0-4940.g 198 kB/s | 23 kB 00:00 2023-07-14T18:52:43.483 INFO:teuthology.orchestra.run.smithi102.stdout:(94/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 9.4 MB/s | 644 kB 00:00 2023-07-14T18:52:43.509 INFO:teuthology.orchestra.run.smithi102.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.5 MB/s | 171 kB 00:00 2023-07-14T18:52:43.511 INFO:teuthology.orchestra.run.smithi078.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-07-14T18:52:43.534 INFO:teuthology.orchestra.run.smithi102.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.7 MB/s | 138 kB 00:00 2023-07-14T18:52:43.592 INFO:teuthology.orchestra.run.smithi102.stdout:(97/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.8 MB/s | 110 kB 00:00 2023-07-14T18:52:43.599 INFO:teuthology.orchestra.run.smithi088.stdout:(28/131): ceph-mgr-cephadm-18.0.0-4940.g81c9387 1.0 MB/s | 129 kB 00:00 2023-07-14T18:52:43.603 INFO:teuthology.orchestra.run.smithi078.stdout:(74/131): socat-1.7.4.1-1.el8.x86_64.rpm 3.4 MB/s | 323 kB 00:00 2023-07-14T18:52:43.643 INFO:teuthology.orchestra.run.smithi102.stdout:(98/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-07-14T18:52:43.687 INFO:teuthology.orchestra.run.smithi078.stdout:(75/131): python3-markupsafe-0.23-19.el8.x86_64 465 kB/s | 39 kB 00:00 2023-07-14T18:52:43.693 INFO:teuthology.orchestra.run.smithi102.stdout:(99/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-07-14T18:52:43.752 INFO:teuthology.orchestra.run.smithi102.stdout:(100/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-07-14T18:52:43.771 INFO:teuthology.orchestra.run.smithi078.stdout:(76/131): ceph-mgr-diskprediction-local-18.0.0- 1.5 MB/s | 7.4 MB 00:04 2023-07-14T18:52:43.813 INFO:teuthology.orchestra.run.smithi078.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 5.3 MB/s | 688 kB 00:00 2023-07-14T18:52:43.895 INFO:teuthology.orchestra.run.smithi102.stdout:(101/131): python3-scipy-1.0.0-21.module+el8.5. 36 MB/s | 14 MB 00:00 2023-07-14T18:52:43.910 INFO:teuthology.orchestra.run.smithi102.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 245 kB/s | 39 kB 00:00 2023-07-14T18:52:43.963 INFO:teuthology.orchestra.run.smithi078.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 2.7 MB/s | 538 kB 00:00 2023-07-14T18:52:43.988 INFO:teuthology.orchestra.run.smithi078.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 925 kB/s | 157 kB 00:00 2023-07-14T18:52:44.002 INFO:teuthology.orchestra.run.smithi102.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 6.2 MB/s | 688 kB 00:00 2023-07-14T18:52:44.070 INFO:teuthology.orchestra.run.smithi102.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 3.3 MB/s | 538 kB 00:00 2023-07-14T18:52:44.141 INFO:teuthology.orchestra.run.smithi088.stdout:(29/131): ceph-mgr-dashboard-18.0.0-4940.g81c93 3.2 MB/s | 1.7 MB 00:00 2023-07-14T18:52:44.155 INFO:teuthology.orchestra.run.smithi078.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 977 kB/s | 187 kB 00:00 2023-07-14T18:52:44.197 INFO:teuthology.orchestra.run.smithi078.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 3.3 MB/s | 713 kB 00:00 2023-07-14T18:52:44.306 INFO:teuthology.orchestra.run.smithi078.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 245 kB/s | 27 kB 00:00 2023-07-14T18:52:44.381 INFO:teuthology.orchestra.run.smithi078.stdout:(83/131): python3-babel-2.5.1-7.el8.noarch.rpm 21 MB/s | 4.8 MB 00:00 2023-07-14T18:52:44.428 INFO:teuthology.orchestra.run.smithi102.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 369 kB/s | 157 kB 00:00 2023-07-14T18:52:44.454 INFO:teuthology.orchestra.run.smithi102.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 488 kB/s | 187 kB 00:00 2023-07-14T18:52:44.490 INFO:teuthology.orchestra.run.smithi078.stdout:(84/131): flex-2.6.1-9.el8.x86_64.rpm 1.7 MB/s | 320 kB 00:00 2023-07-14T18:52:44.554 INFO:teuthology.orchestra.run.smithi102.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 5.6 MB/s | 713 kB 00:00 2023-07-14T18:52:44.598 INFO:teuthology.orchestra.run.smithi078.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 203 kB/s | 44 kB 00:00 2023-07-14T18:52:44.608 INFO:teuthology.orchestra.run.smithi088.stdout:(30/131): ceph-osd-18.0.0-4940.g81c9387d.el8.x8 1.3 MB/s | 18 MB 00:13 2023-07-14T18:52:44.629 INFO:teuthology.orchestra.run.smithi102.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 354 kB/s | 27 kB 00:00 2023-07-14T18:52:44.657 INFO:teuthology.orchestra.run.smithi078.stdout:(86/131): python3-werkzeug-0.12.2-4.el8.noarch. 7.7 MB/s | 457 kB 00:00 2023-07-14T18:52:44.713 INFO:teuthology.orchestra.run.smithi102.stdout:(109/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-07-14T18:52:44.724 INFO:teuthology.orchestra.run.smithi078.stdout:(87/131): python3-numpy-1.14.3-10.el8.x86_64.rp 16 MB/s | 3.7 MB 00:00 2023-07-14T18:52:44.738 INFO:teuthology.orchestra.run.smithi102.stdout:(110/131): flex-2.6.1-9.el8.x86_64.rpm 2.9 MB/s | 320 kB 00:00 2023-07-14T18:52:44.783 INFO:teuthology.orchestra.run.smithi078.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 1.8 MB/s | 225 kB 00:00 2023-07-14T18:52:44.805 INFO:teuthology.orchestra.run.smithi102.stdout:(111/131): python3-prettytable-0.7.2-14.el8.noa 478 kB/s | 44 kB 00:00 2023-07-14T18:52:44.816 INFO:teuthology.orchestra.run.smithi078.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 697 kB/s | 64 kB 00:00 2023-07-14T18:52:44.850 INFO:teuthology.orchestra.run.smithi078.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.8 MB/s | 126 kB 00:00 2023-07-14T18:52:44.855 INFO:teuthology.orchestra.run.smithi102.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 8.9 MB/s | 457 kB 00:00 2023-07-14T18:52:44.875 INFO:teuthology.orchestra.run.smithi078.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 342 kB/s | 20 kB 00:00 2023-07-14T18:52:44.883 INFO:teuthology.orchestra.run.smithi088.stdout:(31/131): ceph-mgr-modules-core-18.0.0-4940.g81 877 kB/s | 241 kB 00:00 2023-07-14T18:52:44.909 INFO:teuthology.orchestra.run.smithi078.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.7 MB/s | 103 kB 00:00 2023-07-14T18:52:44.939 INFO:teuthology.orchestra.run.smithi102.stdout:(113/131): python3-numpy-1.14.3-10.el8.x86_64.r 19 MB/s | 3.7 MB 00:00 2023-07-14T18:52:44.943 INFO:teuthology.orchestra.run.smithi078.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-07-14T18:52:44.964 INFO:teuthology.orchestra.run.smithi102.stdout:(114/131): python3-numpy-f2py-1.14.3-10.el8.x86 2.0 MB/s | 225 kB 00:00 2023-07-14T18:52:44.989 INFO:teuthology.orchestra.run.smithi102.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2023-07-14T18:52:44.992 INFO:teuthology.orchestra.run.smithi088.stdout:(32/131): ceph-mgr-rook-18.0.0-4940.g81c9387d.e 438 kB/s | 47 kB 00:00 2023-07-14T18:52:45.001 INFO:teuthology.orchestra.run.smithi078.stdout:(94/131): libpq-13.5-1.el8.x86_64.rpm 3.3 MB/s | 198 kB 00:00 2023-07-14T18:52:45.015 INFO:teuthology.orchestra.run.smithi102.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-07-14T18:52:45.040 INFO:teuthology.orchestra.run.smithi102.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 398 kB/s | 20 kB 00:00 2023-07-14T18:52:45.065 INFO:teuthology.orchestra.run.smithi102.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-07-14T18:52:45.093 INFO:teuthology.orchestra.run.smithi078.stdout:(95/131): openblas-threads-0.3.15-3.el8.x86_64. 26 MB/s | 4.9 MB 00:00 2023-07-14T18:52:45.099 INFO:teuthology.orchestra.run.smithi102.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-07-14T18:52:45.101 INFO:teuthology.orchestra.run.smithi088.stdout:(33/131): ceph-prometheus-alerts-18.0.0-4940.g8 134 kB/s | 15 kB 00:00 2023-07-14T18:52:45.152 INFO:teuthology.orchestra.run.smithi078.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2023-07-14T18:52:45.158 INFO:teuthology.orchestra.run.smithi102.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 3.3 MB/s | 198 kB 00:00 2023-07-14T18:52:45.219 INFO:teuthology.orchestra.run.smithi078.stdout:(97/131): openblas-0.3.15-3.el8.x86_64.rpm 22 MB/s | 4.7 MB 00:00 2023-07-14T18:52:45.242 INFO:teuthology.orchestra.run.smithi102.stdout:(121/131): openblas-threads-0.3.15-3.el8.x86_64 28 MB/s | 4.9 MB 00:00 2023-07-14T18:52:45.244 INFO:teuthology.orchestra.run.smithi078.stdout:(98/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.8 MB/s | 354 kB 00:00 2023-07-14T18:52:45.300 INFO:teuthology.orchestra.run.smithi102.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 3.4 MB/s | 202 kB 00:00 2023-07-14T18:52:45.320 INFO:teuthology.orchestra.run.smithi078.stdout:(99/131): postgresql-contrib-10.21-2.module+el8 11 MB/s | 808 kB 00:00 2023-07-14T18:52:45.367 INFO:teuthology.orchestra.run.smithi102.stdout:(123/131): openblas-0.3.15-3.el8.x86_64.rpm 23 MB/s | 4.7 MB 00:00 2023-07-14T18:52:45.392 INFO:teuthology.orchestra.run.smithi102.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 3.8 MB/s | 354 kB 00:00 2023-07-14T18:52:45.409 INFO:teuthology.orchestra.run.smithi088.stdout:(34/131): ceph-volume-18.0.0-4940.g81c9387d.el8 838 kB/s | 258 kB 00:00 2023-07-14T18:52:45.436 INFO:teuthology.orchestra.run.smithi078.stdout:(100/131): postgresql-server-10.21-2.module+el8 23 MB/s | 5.1 MB 00:00 2023-07-14T18:52:45.451 INFO:teuthology.orchestra.run.smithi102.stdout:(125/131): postgresql-contrib-10.21-2.module+el 14 MB/s | 808 kB 00:00 2023-07-14T18:52:45.512 INFO:teuthology.orchestra.run.smithi078.stdout:(101/131): postgresql-10.21-2.module+el8.6.0+15 7.8 MB/s | 1.5 MB 00:00 2023-07-14T18:52:45.538 INFO:teuthology.orchestra.run.smithi078.stdout:(102/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 465 kB/s | 47 kB 00:00 2023-07-14T18:52:45.559 INFO:teuthology.orchestra.run.smithi102.stdout:(126/131): postgresql-server-10.21-2.module+el8 26 MB/s | 5.1 MB 00:00 2023-07-14T18:52:45.563 INFO:teuthology.orchestra.run.smithi078.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 3.9 MB/s | 200 kB 00:00 2023-07-14T18:52:45.588 INFO:teuthology.orchestra.run.smithi078.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 849 kB/s | 43 kB 00:00 2023-07-14T18:52:45.601 INFO:teuthology.orchestra.run.smithi102.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 10 MB/s | 1.5 MB 00:00 2023-07-14T18:52:45.613 INFO:teuthology.orchestra.run.smithi078.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-07-14T18:52:45.618 INFO:teuthology.orchestra.run.smithi088.stdout:(35/131): cephadm-18.0.0-4940.g81c9387d.el8.noa 794 kB/s | 165 kB 00:00 2023-07-14T18:52:45.639 INFO:teuthology.orchestra.run.smithi078.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 959 kB/s | 48 kB 00:00 2023-07-14T18:52:45.656 INFO:teuthology.orchestra.run.smithi078.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-07-14T18:52:45.668 INFO:teuthology.orchestra.run.smithi088.stdout:(36/131): rbd-mirror-18.0.0-4940.g81c9387d.el8. 1.1 MB/s | 3.5 MB 00:03 2023-07-14T18:52:45.681 INFO:teuthology.orchestra.run.smithi078.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 1.0 MB/s | 43 kB 00:00 2023-07-14T18:52:45.714 INFO:teuthology.orchestra.run.smithi078.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 1.8 MB/s | 109 kB 00:00 2023-07-14T18:52:45.726 INFO:teuthology.orchestra.run.smithi088.stdout:(37/131): cephfs-top-18.0.0-4940.g81c9387d.el8. 165 kB/s | 18 kB 00:00 2023-07-14T18:52:45.740 INFO:teuthology.orchestra.run.smithi078.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 3.8 MB/s | 229 kB 00:00 2023-07-14T18:52:45.774 INFO:teuthology.orchestra.run.smithi078.stdout:(111/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.3 MB/s | 560 kB 00:00 2023-07-14T18:52:45.791 INFO:teuthology.orchestra.run.smithi078.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 371 kB/s | 19 kB 00:00 2023-07-14T18:52:45.816 INFO:teuthology.orchestra.run.smithi078.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.6 MB/s | 155 kB 00:00 2023-07-14T18:52:45.850 INFO:teuthology.orchestra.run.smithi078.stdout:(114/131): python3-cryptography-3.2.1-5.el8.x86 9.3 MB/s | 559 kB 00:00 2023-07-14T18:52:45.867 INFO:teuthology.orchestra.run.smithi078.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 354 kB/s | 18 kB 00:00 2023-07-14T18:52:45.900 INFO:teuthology.orchestra.run.smithi078.stdout:(116/131): pcre2-devel-10.32-2.el8.x86_64.rpm 12 MB/s | 605 kB 00:00 2023-07-14T18:52:45.953 INFO:teuthology.orchestra.run.smithi078.stdout:(117/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.7 MB/s | 238 kB 00:00 2023-07-14T18:52:46.001 INFO:teuthology.orchestra.run.smithi078.stdout:(118/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.5 MB/s | 72 kB 00:00 2023-07-14T18:52:46.051 INFO:teuthology.orchestra.run.smithi078.stdout:(119/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 15 MB/s | 2.3 MB 00:00 2023-07-14T18:52:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.1 MB/s | 83 kB 00:00 2023-07-14T18:52:46.102 INFO:teuthology.orchestra.run.smithi078.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.2 MB/s | 60 kB 00:00 2023-07-14T18:52:46.127 INFO:teuthology.orchestra.run.smithi088.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 147 kB/s | 59 kB 00:00 2023-07-14T18:52:46.127 INFO:teuthology.orchestra.run.smithi078.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-07-14T18:52:46.152 INFO:teuthology.orchestra.run.smithi078.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-07-14T18:52:46.178 INFO:teuthology.orchestra.run.smithi078.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 765 kB/s | 39 kB 00:00 2023-07-14T18:52:46.211 INFO:teuthology.orchestra.run.smithi078.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 11 MB/s | 644 kB 00:00 2023-07-14T18:52:46.237 INFO:teuthology.orchestra.run.smithi078.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 171 kB 00:00 2023-07-14T18:52:46.254 INFO:teuthology.orchestra.run.smithi078.stdout:(127/131): NetworkManager-initscripts-updown-1. 3.2 MB/s | 138 kB 00:00 2023-07-14T18:52:46.386 INFO:teuthology.orchestra.run.smithi088.stdout:(39/131): ceph-mgr-diskprediction-local-18.0.0- 3.3 MB/s | 7.4 MB 00:02 2023-07-14T18:52:46.411 INFO:teuthology.orchestra.run.smithi088.stdout:(40/131): gperftools-libs-2.7-9.el8.x86_64.rpm 412 kB/s | 306 kB 00:00 2023-07-14T18:52:46.436 INFO:teuthology.orchestra.run.smithi088.stdout:(41/131): libunwind-1.3.1-3.el8.x86_64.rpm 243 kB/s | 75 kB 00:00 2023-07-14T18:52:46.578 INFO:teuthology.orchestra.run.smithi088.stdout:(42/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 308 kB/s | 44 kB 00:00 2023-07-14T18:52:46.703 INFO:teuthology.orchestra.run.smithi088.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 264 kB/s | 33 kB 00:00 2023-07-14T18:52:46.820 INFO:teuthology.orchestra.run.smithi088.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 140 kB/s | 16 kB 00:00 2023-07-14T18:52:46.954 INFO:teuthology.orchestra.run.smithi088.stdout:(45/131): python3-asyncssh-2.7.0-2.el8.noarch.r 838 kB/s | 476 kB 00:00 2023-07-14T18:52:47.012 INFO:teuthology.orchestra.run.smithi088.stdout:(46/131): python3-beautifulsoup4-4.6.3-2.el8.1. 321 kB/s | 185 kB 00:00 2023-07-14T18:52:47.046 INFO:teuthology.orchestra.run.smithi088.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 765 kB/s | 173 kB 00:00 2023-07-14T18:52:47.146 INFO:teuthology.orchestra.run.smithi088.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 609 kB/s | 81 kB 00:00 2023-07-14T18:52:47.180 INFO:teuthology.orchestra.run.smithi088.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-07-14T18:52:47.272 INFO:teuthology.orchestra.run.smithi088.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 140 kB/s | 18 kB 00:00 2023-07-14T18:52:47.305 INFO:teuthology.orchestra.run.smithi088.stdout:(51/131): python3-cherrypy-18.4.0-1.el8.noarch. 1.1 MB/s | 384 kB 00:00 2023-07-14T18:52:47.397 INFO:teuthology.orchestra.run.smithi088.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 389 kB/s | 49 kB 00:00 2023-07-14T18:52:47.422 INFO:teuthology.orchestra.run.smithi088.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 503 kB/s | 59 kB 00:00 2023-07-14T18:52:47.523 INFO:teuthology.orchestra.run.smithi088.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 3.4 MB/s | 1.2 MB 00:00 2023-07-14T18:52:47.548 INFO:teuthology.orchestra.run.smithi088.stdout:(55/131): python3-natsort-7.1.1-2.el8.noarch.rp 400 kB/s | 60 kB 00:00 2023-07-14T18:52:47.752 INFO:teuthology.orchestra.run.smithi088.stdout:(56/131): python3-pecan-1.3.2-9.el8.noarch.rpm 860 kB/s | 283 kB 00:00 2023-07-14T18:52:47.774 INFO:teuthology.orchestra.run.smithi088.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 62 kB/s | 16 kB 00:00 2023-07-14T18:52:47.791 INFO:teuthology.orchestra.run.smithi088.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 139 kB/s | 34 kB 00:00 2023-07-14T18:52:47.908 INFO:teuthology.orchestra.run.smithi088.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 461 kB/s | 61 kB 00:00 2023-07-14T18:52:47.926 INFO:teuthology.orchestra.run.smithi088.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 140 kB/s | 19 kB 00:00 2023-07-14T18:52:47.975 INFO:teuthology.orchestra.run.smithi088.stdout:(61/131): python3-routes-2.4.1-12.el8.noarch.rp 882 kB/s | 196 kB 00:00 2023-07-14T18:52:48.025 INFO:teuthology.orchestra.run.smithi088.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 204 kB/s | 24 kB 00:00 2023-07-14T18:52:48.026 INFO:teuthology.orchestra.run.smithi102.stdout:(128/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.5 MB/s | 3.6 MB 00:02 2023-07-14T18:52:48.042 INFO:teuthology.orchestra.run.smithi088.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 250 kB/s | 29 kB 00:00 2023-07-14T18:52:48.085 INFO:teuthology.orchestra.run.smithi102.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 13 MB/s | 798 kB 00:00 2023-07-14T18:52:48.092 INFO:teuthology.orchestra.run.smithi088.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 235 kB/s | 27 kB 00:00 2023-07-14T18:52:48.159 INFO:teuthology.orchestra.run.smithi088.stdout:(65/131): python3-waitress-1.4.3-1.el8.noarch.r 1.9 MB/s | 255 kB 00:00 2023-07-14T18:52:48.209 INFO:teuthology.orchestra.run.smithi088.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 523 kB/s | 61 kB 00:00 2023-07-14T18:52:48.276 INFO:teuthology.orchestra.run.smithi088.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 732 kB/s | 85 kB 00:00 2023-07-14T18:52:48.302 INFO:teuthology.orchestra.run.smithi088.stdout:(68/131): python3-webob-1.8.5-1.el8.1.noarch.rp 968 kB/s | 251 kB 00:00 2023-07-14T18:52:48.327 INFO:teuthology.orchestra.run.smithi088.stdout:(69/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-07-14T18:52:48.344 INFO:teuthology.orchestra.run.smithi088.stdout:(70/131): python3-zc-lockfile-2.0-2.el8.noarch. 171 kB/s | 23 kB 00:00 2023-07-14T18:52:48.371 INFO:teuthology.orchestra.run.smithi078.stdout:(128/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.7 MB/s | 3.6 MB 00:02 2023-07-14T18:52:48.411 INFO:teuthology.orchestra.run.smithi088.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 561 kB/s | 47 kB 00:00 2023-07-14T18:52:48.436 INFO:teuthology.orchestra.run.smithi088.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 2.1 MB/s | 200 kB 00:00 2023-07-14T18:52:48.437 INFO:teuthology.orchestra.run.smithi078.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 12 MB/s | 798 kB 00:00 2023-07-14T18:52:48.453 INFO:teuthology.orchestra.run.smithi088.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-07-14T18:52:48.477 INFO:teuthology.orchestra.run.smithi102.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 1.3 MB/s | 3.7 MB 00:02 2023-07-14T18:52:48.510 INFO:teuthology.orchestra.run.smithi088.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.9 MB/s | 220 kB 00:00 2023-07-14T18:52:48.537 INFO:teuthology.orchestra.run.smithi088.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 572 kB/s | 48 kB 00:00 2023-07-14T18:52:48.587 INFO:teuthology.orchestra.run.smithi088.stdout:(76/131): thrift-0.13.0-2.el8.x86_64.rpm 5.6 MB/s | 1.7 MB 00:00 2023-07-14T18:52:48.612 INFO:teuthology.orchestra.run.smithi088.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 569 kB/s | 43 kB 00:00 2023-07-14T18:52:48.646 INFO:teuthology.orchestra.run.smithi088.stdout:(78/131): libsepol-devel-2.9-3.el8.x86_64.rpm 640 kB/s | 87 kB 00:00 2023-07-14T18:52:48.672 INFO:teuthology.orchestra.run.smithi088.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.2 MB/s | 229 kB 00:00 2023-07-14T18:52:48.680 INFO:teuthology.orchestra.run.smithi078.stdout:(130/131): librados2-18.0.0-4940.g81c9387d.el8. 1.5 MB/s | 3.7 MB 00:02 2023-07-14T18:52:48.697 INFO:teuthology.orchestra.run.smithi088.stdout:(80/131): python3-pycparser-2.14-14.el8.noarch. 1.0 MB/s | 109 kB 00:00 2023-07-14T18:52:48.722 INFO:teuthology.orchestra.run.smithi088.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 372 kB/s | 19 kB 00:00 2023-07-14T18:52:48.748 INFO:teuthology.orchestra.run.smithi088.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 3.0 MB/s | 155 kB 00:00 2023-07-14T18:52:48.781 INFO:teuthology.orchestra.run.smithi088.stdout:(83/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.1 MB/s | 560 kB 00:00 2023-07-14T18:52:48.806 INFO:teuthology.orchestra.run.smithi088.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 305 kB/s | 18 kB 00:00 2023-07-14T18:52:48.840 INFO:teuthology.orchestra.run.smithi088.stdout:(85/131): python3-cryptography-3.2.1-5.el8.x86_ 4.7 MB/s | 559 kB 00:00 2023-07-14T18:52:48.899 INFO:teuthology.orchestra.run.smithi088.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.5 MB/s | 238 kB 00:00 2023-07-14T18:52:48.924 INFO:teuthology.orchestra.run.smithi088.stdout:(87/131): pcre2-devel-10.32-2.el8.x86_64.rpm 4.1 MB/s | 605 kB 00:00 2023-07-14T18:52:48.958 INFO:teuthology.orchestra.run.smithi088.stdout:(88/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.2 MB/s | 72 kB 00:00 2023-07-14T18:52:48.975 INFO:teuthology.orchestra.run.smithi088.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-07-14T18:52:49.008 INFO:teuthology.orchestra.run.smithi088.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.2 MB/s | 60 kB 00:00 2023-07-14T18:52:49.034 INFO:teuthology.orchestra.run.smithi088.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 1.6 MB/s | 98 kB 00:00 2023-07-14T18:52:49.059 INFO:teuthology.orchestra.run.smithi088.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2023-07-14T18:52:49.084 INFO:teuthology.orchestra.run.smithi088.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 770 kB/s | 39 kB 00:00 2023-07-14T18:52:49.142 INFO:teuthology.orchestra.run.smithi088.stdout:(94/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 7.7 MB/s | 2.3 MB 00:00 2023-07-14T18:52:49.168 INFO:teuthology.orchestra.run.smithi088.stdout:(95/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 5.8 MB/s | 644 kB 00:00 2023-07-14T18:52:49.202 INFO:teuthology.orchestra.run.smithi088.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.3 MB/s | 138 kB 00:00 2023-07-14T18:52:49.250 INFO:teuthology.orchestra.run.smithi088.stdout:(97/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 1.0 MB/s | 171 kB 00:00 2023-07-14T18:52:49.277 INFO:teuthology.orchestra.run.smithi088.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.4 MB/s | 110 kB 00:00 2023-07-14T18:52:49.302 INFO:teuthology.orchestra.run.smithi088.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.0 MB/s | 54 kB 00:00 2023-07-14T18:52:49.328 INFO:teuthology.orchestra.run.smithi088.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2023-07-14T18:52:49.361 INFO:teuthology.orchestra.run.smithi088.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-07-14T18:52:49.378 INFO:teuthology.orchestra.run.smithi088.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 775 kB/s | 39 kB 00:00 2023-07-14T18:52:49.420 INFO:teuthology.orchestra.run.smithi088.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 11 MB/s | 688 kB 00:00 2023-07-14T18:52:49.462 INFO:teuthology.orchestra.run.smithi088.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 6.3 MB/s | 538 kB 00:00 2023-07-14T18:52:49.487 INFO:teuthology.orchestra.run.smithi088.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.3 MB/s | 157 kB 00:00 2023-07-14T18:52:49.513 INFO:teuthology.orchestra.run.smithi088.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-07-14T18:52:49.555 INFO:teuthology.orchestra.run.smithi088.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-07-14T18:52:49.588 INFO:teuthology.orchestra.run.smithi088.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 797 kB/s | 27 kB 00:00 2023-07-14T18:52:49.647 INFO:teuthology.orchestra.run.smithi088.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 5.4 MB/s | 320 kB 00:00 2023-07-14T18:52:49.689 INFO:teuthology.orchestra.run.smithi088.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.0 MB/s | 44 kB 00:00 2023-07-14T18:52:49.756 INFO:teuthology.orchestra.run.smithi088.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 20 MB/s | 4.8 MB 00:00 2023-07-14T18:52:49.806 INFO:teuthology.orchestra.run.smithi088.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 9.0 MB/s | 457 kB 00:00 2023-07-14T18:52:49.848 INFO:teuthology.orchestra.run.smithi088.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-07-14T18:52:49.907 INFO:teuthology.orchestra.run.smithi088.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 17 MB/s | 3.7 MB 00:00 2023-07-14T18:52:49.932 INFO:teuthology.orchestra.run.smithi088.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 767 kB/s | 64 kB 00:00 2023-07-14T18:52:49.957 INFO:teuthology.orchestra.run.smithi088.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.5 MB/s | 126 kB 00:00 2023-07-14T18:52:50.016 INFO:teuthology.orchestra.run.smithi088.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 239 kB/s | 20 kB 00:00 2023-07-14T18:52:50.032 INFO:teuthology.orchestra.run.smithi088.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 1.3 MB/s | 103 kB 00:00 2023-07-14T18:52:50.075 INFO:teuthology.orchestra.run.smithi088.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-07-14T18:52:50.225 INFO:teuthology.orchestra.run.smithi088.stdout:(120/131): python3-scipy-1.0.0-21.module+el8.5. 13 MB/s | 14 MB 00:01 2023-07-14T18:52:50.250 INFO:teuthology.orchestra.run.smithi088.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 1.1 MB/s | 198 kB 00:00 2023-07-14T18:52:50.300 INFO:teuthology.orchestra.run.smithi088.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.0 MB/s | 202 kB 00:00 2023-07-14T18:52:50.417 INFO:teuthology.orchestra.run.smithi088.stdout:(123/131): openblas-threads-0.3.15-3.el8.x86_64 13 MB/s | 4.9 MB 00:00 2023-07-14T18:52:50.442 INFO:teuthology.orchestra.run.smithi088.stdout:(124/131): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-07-14T18:52:50.518 INFO:teuthology.orchestra.run.smithi088.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 16 MB/s | 4.7 MB 00:00 2023-07-14T18:52:50.552 INFO:teuthology.orchestra.run.smithi088.stdout:(126/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-07-14T18:52:50.602 INFO:teuthology.orchestra.run.smithi088.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 18 MB/s | 1.5 MB 00:00 2023-07-14T18:52:50.677 INFO:teuthology.orchestra.run.smithi088.stdout:(128/131): postgresql-server-10.21-2.module+el8 20 MB/s | 5.1 MB 00:00 2023-07-14T18:52:50.752 INFO:teuthology.orchestra.run.smithi088.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 10 MB/s | 798 kB 00:00 2023-07-14T18:52:53.086 INFO:teuthology.orchestra.run.smithi088.stdout:(130/131): librbd1-18.0.0-4940.g81c9387d.el8.x8 1.5 MB/s | 3.6 MB 00:02 2023-07-14T18:52:53.302 INFO:teuthology.orchestra.run.smithi088.stdout:(131/131): librados2-18.0.0-4940.g81c9387d.el8. 1.3 MB/s | 3.7 MB 00:02 2023-07-14T18:52:53.305 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:52:53.306 INFO:teuthology.orchestra.run.smithi088.stdout:Total 7.3 MB/s | 206 MB 00:28 2023-07-14T18:52:54.144 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2023-07-14T18:52:54.264 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2023-07-14T18:52:54.265 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2023-07-14T18:52:55.382 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2023-07-14T18:52:55.386 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2023-07-14T18:52:55.413 INFO:teuthology.orchestra.run.smithi078.stdout:(131/131): ceph-test-18.0.0-4940.g81c9387d.el8. 2.1 MB/s | 48 MB 00:23 2023-07-14T18:52:55.416 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:52:55.416 INFO:teuthology.orchestra.run.smithi078.stdout:Total 6.5 MB/s | 206 MB 00:31 2023-07-14T18:52:55.585 INFO:teuthology.orchestra.run.smithi102.stdout:(131/131): ceph-test-18.0.0-4940.g81c9387d.el8. 1.6 MB/s | 48 MB 00:29 2023-07-14T18:52:55.590 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:52:55.590 INFO:teuthology.orchestra.run.smithi102.stdout:Total 6.2 MB/s | 206 MB 00:33 2023-07-14T18:52:56.220 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction check 2023-07-14T18:52:56.341 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction check succeeded. 2023-07-14T18:52:56.341 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction test 2023-07-14T18:52:56.399 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2023-07-14T18:52:56.518 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2023-07-14T18:52:56.518 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2023-07-14T18:52:57.451 INFO:teuthology.orchestra.run.smithi078.stdout:Transaction test succeeded. 2023-07-14T18:52:57.455 INFO:teuthology.orchestra.run.smithi078.stdout:Running transaction 2023-07-14T18:52:57.624 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2023-07-14T18:52:57.628 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2023-07-14T18:53:00.405 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2023-07-14T18:53:00.609 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:53:00.817 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:53:01.011 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T18:53:01.123 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T18:53:01.352 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T18:53:01.754 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T18:53:01.960 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T18:53:02.406 INFO:teuthology.orchestra.run.smithi078.stdout: Preparing : 1/1 2023-07-14T18:53:02.408 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T18:53:02.477 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2023-07-14T18:53:02.577 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:53:02.591 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:53:02.643 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-07-14T18:53:02.692 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:53:02.806 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:53:02.818 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-07-14T18:53:02.894 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:53:02.990 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:53:03.012 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T18:53:03.025 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-07-14T18:53:03.117 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-07-14T18:53:03.190 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:53:03.227 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:53:03.320 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T18:53:03.336 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-07-14T18:53:03.352 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T18:53:03.598 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:53:03.619 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:53:03.720 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-07-14T18:53:03.788 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:53:03.928 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T18:53:04.016 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-07-14T18:53:04.191 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:53:04.252 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:53:04.328 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 5/134 2023-07-14T18:53:04.447 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:53:04.487 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:53:04.568 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:53:04.601 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:53:04.650 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:53:04.711 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:53:04.779 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:53:04.827 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:53:04.852 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:53:04.873 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:53:04.937 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:53:04.996 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:53:05.033 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:53:05.152 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:53:05.202 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-07-14T18:53:05.221 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T18:53:05.251 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:53:05.432 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:53:05.432 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T18:53:05.449 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:53:05.620 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T18:53:05.648 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:53:05.705 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-07-14T18:53:05.767 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T18:53:05.896 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-07-14T18:53:05.911 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T18:53:06.050 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:53:06.068 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T18:53:06.282 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:53:06.345 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-07-14T18:53:06.411 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:53:06.507 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:53:06.645 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:53:06.707 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T18:53:06.771 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:53:06.813 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-07-14T18:53:06.909 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T18:53:06.973 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-07-14T18:53:06.984 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:53:07.093 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 14/134 2023-07-14T18:53:07.183 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-07-14T18:53:07.243 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:53:07.321 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 15/134 2023-07-14T18:53:07.351 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-07-14T18:53:07.493 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:53:07.500 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T18:53:07.503 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T18:53:07.620 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-07-14T18:53:07.645 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-07-14T18:53:07.692 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T18:53:07.776 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:53:07.803 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T18:53:08.372 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-07-14T18:53:08.382 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T18:53:08.418 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-07-14T18:53:08.560 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:53:08.561 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T18:53:08.615 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T18:53:08.831 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T18:53:09.069 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:53:09.154 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-07-14T18:53:09.159 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-07-14T18:53:09.265 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:53:09.319 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T18:53:09.395 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:53:09.401 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:53:09.656 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:53:09.842 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T18:53:10.015 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-07-14T18:53:10.095 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-07-14T18:53:10.268 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-07-14T18:53:10.366 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-07-14T18:53:10.449 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-07-14T18:53:10.594 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-07-14T18:53:10.679 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:53:10.846 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-07-14T18:53:10.859 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:53:10.980 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:53:11.048 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-07-14T18:53:11.248 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:53:11.290 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-07-14T18:53:11.452 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T18:53:11.527 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-07-14T18:53:11.623 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T18:53:11.716 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 26/134 2023-07-14T18:53:11.828 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 27/134 2023-07-14T18:53:12.627 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T18:53:12.773 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 28/134 2023-07-14T18:53:12.963 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T18:53:13.074 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-07-14T18:53:13.202 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-07-14T18:53:13.256 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T18:53:13.353 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-07-14T18:53:13.500 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T18:53:13.619 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-07-14T18:53:13.744 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:53:13.809 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:53:14.204 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-07-14T18:53:14.243 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:53:14.413 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:53:14.515 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-07-14T18:53:14.647 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:53:14.787 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-07-14T18:53:14.798 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T18:53:14.967 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T18:53:14.987 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-07-14T18:53:15.132 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T18:53:15.213 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:53:15.218 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T18:53:15.387 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:53:15.437 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T18:53:15.557 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:53:15.587 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-07-14T18:53:15.694 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T18:53:15.725 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:53:15.841 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T18:53:15.884 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-07-14T18:53:15.915 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T18:53:15.935 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-07-14T18:53:16.077 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-07-14T18:53:16.145 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T18:53:16.145 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-07-14T18:53:16.285 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T18:53:16.302 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T18:53:16.347 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-07-14T18:53:16.368 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-07-14T18:53:16.522 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T18:53:16.564 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 36/134 2023-07-14T18:53:16.601 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-07-14T18:53:16.692 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T18:53:16.771 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 37/134 2023-07-14T18:53:16.847 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-07-14T18:53:16.893 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T18:53:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-07-14T18:53:17.010 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T18:53:17.027 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T18:53:17.092 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-07-14T18:53:17.123 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-07-14T18:53:17.254 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T18:53:17.282 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-07-14T18:53:17.327 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-07-14T18:53:17.443 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T18:53:17.447 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-07-14T18:53:17.470 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-07-14T18:53:17.610 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-07-14T18:53:17.766 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-07-14T18:53:17.795 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-07-14T18:53:17.816 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T18:53:17.965 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-07-14T18:53:17.971 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-07-14T18:53:18.113 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-07-14T18:53:18.117 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T18:53:18.342 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-07-14T18:53:18.360 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T18:53:18.391 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-07-14T18:53:18.540 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-07-14T18:53:18.587 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T18:53:18.720 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-07-14T18:53:18.786 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T18:53:18.905 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-07-14T18:53:18.934 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-07-14T18:53:19.130 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T18:53:19.193 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-07-14T18:53:19.211 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-07-14T18:53:19.373 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T18:53:19.421 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-07-14T18:53:19.427 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-07-14T18:53:19.557 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T18:53:19.629 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-07-14T18:53:19.741 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-07-14T18:53:19.803 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T18:53:19.864 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-07-14T18:53:19.932 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-07-14T18:53:19.960 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T18:53:20.151 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-07-14T18:53:20.154 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T18:53:20.166 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-07-14T18:53:20.359 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-07-14T18:53:20.395 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-07-14T18:53:20.472 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T18:53:20.552 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-07-14T18:53:20.595 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-07-14T18:53:20.752 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T18:53:20.766 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-07-14T18:53:20.772 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-07-14T18:53:20.938 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-07-14T18:53:21.003 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T18:53:21.104 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-07-14T18:53:21.116 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-07-14T18:53:21.249 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T18:53:21.371 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-07-14T18:53:21.400 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-07-14T18:53:21.406 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:53:21.618 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:53:21.630 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-07-14T18:53:21.645 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-07-14T18:53:21.867 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:53:21.889 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-07-14T18:53:21.912 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-07-14T18:53:22.090 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-07-14T18:53:22.102 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-07-14T18:53:22.121 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:53:22.255 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-07-14T18:53:22.267 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:53:22.268 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:53:22.420 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:53:22.434 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:53:22.457 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:53:22.543 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-07-14T18:53:22.609 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:53:22.677 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:53:22.800 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-07-14T18:53:22.828 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:53:22.883 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:53:22.993 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:53:23.022 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:53:23.027 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-07-14T18:53:23.178 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:53:23.183 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:53:23.209 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:53:23.262 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-07-14T18:53:23.362 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:53:23.383 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:53:23.529 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-07-14T18:53:23.594 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:53:23.603 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:53:23.717 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-07-14T18:53:23.777 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:53:23.827 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:53:23.916 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-07-14T18:53:23.928 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:53:24.037 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:53:24.073 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:53:24.104 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-07-14T18:53:24.280 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:53:24.301 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:53:24.414 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-07-14T18:53:24.538 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:53:24.545 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:53:24.623 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 70/134 2023-07-14T18:53:24.723 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:53:24.747 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:53:24.804 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-07-14T18:53:24.945 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:53:24.952 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-07-14T18:53:24.974 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:53:25.133 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:53:25.150 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-07-14T18:53:25.265 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:53:25.387 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-07-14T18:53:25.463 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:53:25.561 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-07-14T18:53:25.760 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-07-14T18:53:25.905 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-07-14T18:53:26.066 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:53:26.111 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-07-14T18:53:26.336 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-07-14T18:53:26.342 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:53:26.521 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-07-14T18:53:26.544 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:53:26.605 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:53:26.794 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-07-14T18:53:26.837 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:53:26.973 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:53:27.091 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:53:27.099 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-07-14T18:53:27.263 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:53:27.264 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-07-14T18:53:27.401 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:53:27.465 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-07-14T18:53:27.477 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:53:27.676 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:53:27.892 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:53:28.272 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:53:28.454 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:53:28.528 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:53:28.579 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-07-14T18:53:28.671 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:53:28.685 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:53:28.842 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:53:28.852 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-07-14T18:53:28.915 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:53:29.092 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:53:29.098 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-07-14T18:53:29.450 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-07-14T18:53:29.724 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-07-14T18:53:29.995 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-07-14T18:53:30.784 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-07-14T18:53:30.956 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:53:31.026 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 92/134 2023-07-14T18:53:31.257 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 93/134 2023-07-14T18:53:32.434 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:32.646 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:32.791 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:33.025 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:33.226 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:33.478 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:53:33.565 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:53:33.606 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:33.703 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:53:33.901 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:53:33.983 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:53:34.124 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:53:34.861 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:35.073 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:35.667 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 94/134 2023-07-14T18:53:35.996 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:53:36.078 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 95/134 2023-07-14T18:53:36.239 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:53:52.254 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:53:52.254 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2023-07-14T18:53:52.255 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2023-07-14T18:53:52.255 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2023-07-14T18:53:52.255 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /run 2023-07-14T18:53:52.255 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2023-07-14T18:53:52.255 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2023-07-14T18:53:52.255 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2023-07-14T18:53:52.255 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2023-07-14T18:53:52.255 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /dev 2023-07-14T18:53:52.256 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:53:52.621 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:53:52.621 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2023-07-14T18:53:52.621 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2023-07-14T18:53:52.622 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /dev 2023-07-14T18:53:52.622 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /run 2023-07-14T18:53:52.622 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2023-07-14T18:53:52.622 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2023-07-14T18:53:52.622 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2023-07-14T18:53:52.622 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2023-07-14T18:53:52.622 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /dev 2023-07-14T18:53:52.622 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:53:52.624 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:53:52.997 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:53:54.203 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:53:54.424 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:53:54.581 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:53:54.642 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:53:54.790 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:53:54.824 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:53:55.009 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:53:55.055 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 96/134 2023-07-14T18:53:55.055 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /sys 2023-07-14T18:53:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /proc 2023-07-14T18:53:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2023-07-14T18:53:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /run 2023-07-14T18:53:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /mnt 2023-07-14T18:53:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /var/tmp 2023-07-14T18:53:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /home 2023-07-14T18:53:55.056 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /tmp 2023-07-14T18:53:55.057 INFO:teuthology.orchestra.run.smithi078.stdout:skipping the directory /dev 2023-07-14T18:53:55.057 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:53:55.197 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:53:55.277 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:53:55.404 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:53:55.482 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:53:55.499 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:53:55.650 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:53:55.813 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:53:55.830 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:53:56.984 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 97/134 2023-07-14T18:53:57.092 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:53:57.184 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:53:57.284 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:53:57.396 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:53:57.402 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 98/134 2023-07-14T18:53:57.501 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:53:57.578 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:53:57.588 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:53:57.667 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:53:57.797 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:53:57.802 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:53:57.994 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:53:58.014 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:53:58.031 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 99/134 2023-07-14T18:53:58.171 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:53:58.270 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:53:58.287 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 100/134 2023-07-14T18:53:58.289 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:53:58.386 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:53:58.664 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:53:59.248 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:53:59.393 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:53:59.480 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:53:59.619 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:53:59.813 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:53:59.828 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:53:59.834 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:53:59.902 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:54:00.003 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 101/134 2023-07-14T18:54:00.037 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:54:00.072 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:54:00.167 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 102/134 2023-07-14T18:54:00.250 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:54:00.255 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:54:00.267 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:54:00.350 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 103/134 2023-07-14T18:54:00.455 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:54:00.490 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:54:00.560 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:54:00.637 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:54:00.648 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:54:00.777 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 104/134 2023-07-14T18:54:01.092 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:54:01.563 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:54:01.790 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:54:01.824 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:54:01.956 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:54:01.972 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:54:02.014 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:54:02.051 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 105/134 2023-07-14T18:54:02.151 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:54:02.184 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:54:02.202 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:54:02.205 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:54:02.293 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 106/134 2023-07-14T18:54:02.321 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:54:02.362 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:54:02.521 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:54:02.557 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:54:02.687 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 107/134 2023-07-14T18:54:02.697 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:54:02.871 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:54:02.950 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:54:02.985 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 108/134 2023-07-14T18:54:03.021 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:54:03.135 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:54:03.146 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 109/134 2023-07-14T18:54:04.441 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 110/134 2023-07-14T18:54:04.653 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:54:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 111/134 2023-07-14T18:54:04.844 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:54:04.886 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-4940.g81c93 112/134 2023-07-14T18:54:05.052 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:54:05.403 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 113/134 2023-07-14T18:54:05.560 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:54:07.249 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:54:07.559 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:54:08.125 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:54:08.425 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:54:08.477 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T18:54:08.623 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T18:54:08.726 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-07-14T18:54:08.898 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 117/134 2023-07-14T18:54:08.903 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T18:54:09.070 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T18:54:09.175 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-07-14T18:54:09.262 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T18:54:09.345 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T18:54:09.449 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T18:54:09.543 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-07-14T18:54:09.649 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:54:09.788 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:54:09.790 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-07-14T18:54:09.965 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:54:10.102 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 114/134 2023-07-14T18:54:10.121 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:54:10.402 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 115/134 2023-07-14T18:54:10.576 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:54:10.781 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-07-14T18:54:11.663 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-07-14T18:54:11.983 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-07-14T18:54:12.028 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:12.212 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 119/134 2023-07-14T18:54:12.286 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:12.306 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:12.315 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:12.405 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:54:12.458 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:54:12.561 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:12.578 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:12.622 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 121/134 2023-07-14T18:54:12.720 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:54:12.777 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:54:12.794 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:54:13.027 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:54:13.154 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:54:13.365 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:54:13.513 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:54:13.677 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T18:54:13.697 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:54:13.768 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T18:54:13.869 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T18:54:13.952 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-07-14T18:54:13.953 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T18:54:13.965 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T18:54:14.144 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T18:54:14.152 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T18:54:14.152 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T18:54:14.164 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-07-14T18:54:14.288 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T18:54:14.288 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:14.336 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T18:54:14.343 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-07-14T18:54:14.513 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-07-14T18:54:14.513 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:14.530 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:14.531 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:54:14.697 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:14.697 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:54:14.768 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:54:14.932 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:54:14.952 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:14.976 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:15.134 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:15.201 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:15.219 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-07-14T18:54:15.455 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 124/134 2023-07-14T18:54:15.482 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:15.482 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:54:15.483 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:54:15.483 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:54:15.483 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:54:15.483 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:54:15.483 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:54:15.483 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:54:15.483 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:54:15.484 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:54:15.484 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:54:15.484 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:54:15.484 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:54:15.484 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:54:15.484 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:54:15.484 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:54:15.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:54:15.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:54:15.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:54:15.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:54:15.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:54:15.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:54:15.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:54:15.485 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:54:15.486 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:54:15.486 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:54:15.486 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:54:15.486 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:54:15.493 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:54:15.493 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:54:15.493 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:54:15.493 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:54:15.494 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:54:15.494 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:54:15.494 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:54:15.494 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:54:15.494 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:54:15.494 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:54:15.494 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:54:15.494 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:54:15.495 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:54:15.495 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:54:15.495 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:54:15.495 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:54:15.495 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:54:15.495 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:54:15.495 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:54:15.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:54:15.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:54:15.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:54:15.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:54:15.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:54:15.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:54:15.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:54:15.496 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:54:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:54:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:54:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:54:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:54:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:54:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:54:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:54:15.497 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:54:15.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:54:15.499 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:54:15.499 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T18:54:15.499 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T18:54:15.499 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T18:54:15.499 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T18:54:15.499 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T18:54:15.499 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T18:54:15.500 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T18:54:15.500 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T18:54:15.500 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T18:54:15.500 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T18:54:15.500 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T18:54:15.500 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T18:54:15.500 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T18:54:15.500 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T18:54:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T18:54:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T18:54:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T18:54:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T18:54:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T18:54:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T18:54:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T18:54:15.501 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T18:54:15.502 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T18:54:15.502 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T18:54:15.502 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T18:54:15.502 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T18:54:15.502 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T18:54:15.502 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T18:54:15.502 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T18:54:15.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T18:54:15.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T18:54:15.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T18:54:15.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T18:54:15.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T18:54:15.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T18:54:15.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T18:54:15.503 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T18:54:15.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T18:54:15.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T18:54:15.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T18:54:15.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T18:54:15.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T18:54:15.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T18:54:15.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T18:54:15.504 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T18:54:15.505 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T18:54:15.505 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T18:54:15.505 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T18:54:15.505 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T18:54:15.505 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T18:54:15.505 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T18:54:15.505 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T18:54:15.505 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T18:54:15.506 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T18:54:15.506 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T18:54:15.506 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T18:54:15.506 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T18:54:15.506 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:54:15.506 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:54:15.507 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:54:15.507 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:15.507 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:54:15.632 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:54:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:54:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:54:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:54:15.633 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:54:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:54:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:54:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:54:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:54:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:54:15.634 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:54:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:54:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:54:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:54:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:54:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:54:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:54:15.635 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:54:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:54:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:54:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:54:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:54:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:54:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:54:15.636 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:54:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:54:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:54:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:54:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:54:15.637 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:54:15.647 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:54:15.647 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:54:15.647 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:54:15.647 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:54:15.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:54:15.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:54:15.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:54:15.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:54:15.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:54:15.648 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:54:15.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:54:15.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:54:15.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:54:15.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:54:15.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:54:15.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:54:15.649 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:54:15.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:54:15.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:54:15.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:54:15.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:54:15.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:54:15.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:54:15.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:54:15.650 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:54:15.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:54:15.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:54:15.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:54:15.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:54:15.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:54:15.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:54:15.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:54:15.651 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:54:15.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:54:15.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:54:15.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:54:15.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:54:15.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:54:15.652 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:54:15.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:54:15.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:54:15.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-07-14T18:54:15.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-07-14T18:54:15.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-07-14T18:54:15.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-07-14T18:54:15.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-07-14T18:54:15.653 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-07-14T18:54:15.654 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-07-14T18:54:15.654 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-07-14T18:54:15.654 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-07-14T18:54:15.654 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-07-14T18:54:15.654 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-07-14T18:54:15.654 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-07-14T18:54:15.654 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-07-14T18:54:15.654 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-07-14T18:54:15.655 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-07-14T18:54:15.655 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-07-14T18:54:15.655 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-07-14T18:54:15.655 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-07-14T18:54:15.655 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-07-14T18:54:15.655 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-07-14T18:54:15.655 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-07-14T18:54:15.656 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-07-14T18:54:15.656 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-07-14T18:54:15.656 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-07-14T18:54:15.656 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-07-14T18:54:15.656 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-07-14T18:54:15.656 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-07-14T18:54:15.656 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-07-14T18:54:15.657 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-07-14T18:54:15.657 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-07-14T18:54:15.657 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-07-14T18:54:15.657 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-07-14T18:54:15.657 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-07-14T18:54:15.657 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-07-14T18:54:15.657 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-07-14T18:54:15.657 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-07-14T18:54:15.658 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-07-14T18:54:15.658 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-07-14T18:54:15.658 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-07-14T18:54:15.658 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-07-14T18:54:15.658 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-07-14T18:54:15.658 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-07-14T18:54:15.658 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-07-14T18:54:15.659 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-07-14T18:54:15.659 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-07-14T18:54:15.659 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-07-14T18:54:15.659 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-07-14T18:54:15.659 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-07-14T18:54:15.659 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-07-14T18:54:15.660 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-07-14T18:54:15.660 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-07-14T18:54:15.660 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-07-14T18:54:15.660 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-07-14T18:54:15.661 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-07-14T18:54:15.661 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-07-14T18:54:15.661 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-07-14T18:54:15.661 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-07-14T18:54:15.662 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:54:15.662 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:54:15.662 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:54:15.662 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:15.663 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:54:15.709 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 125/134 2023-07-14T18:54:16.038 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 126/134 2023-07-14T18:54:16.361 INFO:teuthology.orchestra.run.smithi078.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-07-14T18:54:16.500 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:54:16.509 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:54:16.707 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-07-14T18:54:16.900 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:54:16.985 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-07-14T18:54:17.234 INFO:teuthology.orchestra.run.smithi078.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:54:17.246 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-07-14T18:54:17.247 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:17.446 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:17.446 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:54:17.752 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-07-14T18:54:17.957 INFO:teuthology.orchestra.run.smithi078.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:18.000 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:18.000 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:04 ago on Fri 14 Jul 2023 06:52:13 PM UTC. 2023-07-14T18:54:18.070 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:18.070 INFO:teuthology.orchestra.run.smithi102.stdout:Last metadata expiration check: 0:02:08 ago on Fri 14 Jul 2023 06:52:10 PM UTC. 2023-07-14T18:54:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 1/134 2023-07-14T18:54:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2/134 2023-07-14T18:54:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 3/134 2023-07-14T18:54:18.487 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 4/134 2023-07-14T18:54:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-4940.g81c93 5/134 2023-07-14T18:54:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 6/134 2023-07-14T18:54:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 7/134 2023-07-14T18:54:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 8/134 2023-07-14T18:54:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 9/134 2023-07-14T18:54:18.488 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 10/134 2023-07-14T18:54:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 11/134 2023-07-14T18:54:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 12/134 2023-07-14T18:54:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 13/134 2023-07-14T18:54:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_ 14/134 2023-07-14T18:54:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 15/134 2023-07-14T18:54:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 16/134 2023-07-14T18:54:18.489 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_6 17/134 2023-07-14T18:54:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86 18/134 2023-07-14T18:54:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 19/134 2023-07-14T18:54:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el 20/134 2023-07-14T18:54:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-ceph-common-2:18.0.0-4940.g81c9387d.el8. 21/134 2023-07-14T18:54:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_6 22/134 2023-07-14T18:54:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 23/134 2023-07-14T18:54:18.490 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 24/134 2023-07-14T18:54:18.491 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 25/134 2023-07-14T18:54:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 26/134 2023-07-14T18:54:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 27/134 2023-07-14T18:54:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 28/134 2023-07-14T18:54:18.499 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d. 29/134 2023-07-14T18:54:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noa 30/134 2023-07-14T18:54:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.n 31/134 2023-07-14T18:54:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c 32/134 2023-07-14T18:54:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el 33/134 2023-07-14T18:54:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 34/134 2023-07-14T18:54:18.500 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.e 35/134 2023-07-14T18:54:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 36/134 2023-07-14T18:54:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 37/134 2023-07-14T18:54:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 38/134 2023-07-14T18:54:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-07-14T18:54:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-07-14T18:54:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-07-14T18:54:18.501 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-07-14T18:54:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-07-14T18:54:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-07-14T18:54:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-07-14T18:54:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-07-14T18:54:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-07-14T18:54:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-07-14T18:54:18.502 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-07-14T18:54:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-07-14T18:54:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-07-14T18:54:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-07-14T18:54:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-07-14T18:54:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-07-14T18:54:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-07-14T18:54:18.503 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-07-14T18:54:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-07-14T18:54:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-07-14T18:54:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-07-14T18:54:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-07-14T18:54:18.504 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-07-14T18:54:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-07-14T18:54:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-07-14T18:54:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-07-14T18:54:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/134 2023-07-14T18:54:18.505 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-07-14T18:54:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-07-14T18:54:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-07-14T18:54:18.507 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-07-14T18:54:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-07-14T18:54:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-07-14T18:54:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-07-14T18:54:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-07-14T18:54:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-07-14T18:54:18.508 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-07-14T18:54:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-07-14T18:54:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-07-14T18:54:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-07-14T18:54:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-07-14T18:54:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-07-14T18:54:18.509 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-07-14T18:54:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-07-14T18:54:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-07-14T18:54:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-07-14T18:54:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-07-14T18:54:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-07-14T18:54:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-07-14T18:54:18.510 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-07-14T18:54:18.511 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-07-14T18:54:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-07-14T18:54:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-07-14T18:54:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-07-14T18:54:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-07-14T18:54:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-07-14T18:54:18.512 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-07-14T18:54:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-07-14T18:54:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-07-14T18:54:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-07-14T18:54:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-07-14T18:54:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-07-14T18:54:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-07-14T18:54:18.513 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-07-14T18:54:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-07-14T18:54:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-07-14T18:54:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-07-14T18:54:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-07-14T18:54:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-07-14T18:54:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-07-14T18:54:18.514 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-07-14T18:54:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-07-14T18:54:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-07-14T18:54:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-07-14T18:54:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-07-14T18:54:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-07-14T18:54:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-07-14T18:54:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-07-14T18:54:18.515 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-07-14T18:54:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-07-14T18:54:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-07-14T18:54:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-07-14T18:54:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-07-14T18:54:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-07-14T18:54:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-07-14T18:54:18.516 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-07-14T18:54:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-07-14T18:54:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-07-14T18:54:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-07-14T18:54:18.517 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-07-14T18:54:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 129/134 2023-07-14T18:54:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-07-14T18:54:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 131/134 2023-07-14T18:54:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-07-14T18:54:18.518 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-07-14T18:54:21.042 INFO:teuthology.orchestra.run.smithi088.stdout:Installed products updated. 2023-07-14T18:54:21.065 INFO:teuthology.orchestra.run.smithi102.stdout:Installed products updated. 2023-07-14T18:54:21.087 INFO:teuthology.orchestra.run.smithi078.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-07-14T18:54:21.087 INFO:teuthology.orchestra.run.smithi078.stdout:Last metadata expiration check: 0:02:10 ago on Fri 14 Jul 2023 06:52:11 PM UTC. 2023-07-14T18:54:21.256 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:54:21.256 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2023-07-14T18:54:21.256 INFO:teuthology.orchestra.run.smithi088.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:54:21.257 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.257 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.257 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2023-07-14T18:54:21.257 INFO:teuthology.orchestra.run.smithi088.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:54:21.257 INFO:teuthology.orchestra.run.smithi088.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:54:21.257 INFO:teuthology.orchestra.run.smithi088.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:54:21.257 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.258 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.258 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.258 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.258 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.258 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.258 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.258 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.260 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.260 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.260 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.260 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.260 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.260 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.260 INFO:teuthology.orchestra.run.smithi088.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.260 INFO:teuthology.orchestra.run.smithi088.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:54:21.261 INFO:teuthology.orchestra.run.smithi088.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:54:21.261 INFO:teuthology.orchestra.run.smithi088.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:54:21.261 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:54:21.261 INFO:teuthology.orchestra.run.smithi088.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:54:21.261 INFO:teuthology.orchestra.run.smithi088.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:54:21.261 INFO:teuthology.orchestra.run.smithi088.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:54:21.261 INFO:teuthology.orchestra.run.smithi088.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:54:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:54:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:54:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:54:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:54:21.262 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:54:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:54:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:54:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:54:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:54:21.263 INFO:teuthology.orchestra.run.smithi088.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:54:21.264 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:54:21.264 INFO:teuthology.orchestra.run.smithi088.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:54:21.264 INFO:teuthology.orchestra.run.smithi088.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:54:21.264 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:54:21.264 INFO:teuthology.orchestra.run.smithi088.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:54:21.264 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:54:21.265 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:54:21.265 INFO:teuthology.orchestra.run.smithi088.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:54:21.265 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:54:21.265 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:54:21.265 INFO:teuthology.orchestra.run.smithi088.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:54:21.265 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:21.265 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:21.266 INFO:teuthology.orchestra.run.smithi088.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:21.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:54:21.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:54:21.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:54:21.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:54:21.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:54:21.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:54:21.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:54:21.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:54:21.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:54:21.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:54:21.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:54:21.268 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:54:21.268 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:54:21.268 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:54:21.268 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:54:21.268 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:54:21.268 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:54:21.268 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:54:21.268 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:54:21.269 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:54:21.269 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:54:21.269 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:54:21.269 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:54:21.269 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:54:21.269 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:54:21.269 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:54:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:54:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:54:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:54:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:54:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:54:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:54:21.270 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:54:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:54:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:54:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:54:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:54:21.271 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:54:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:54:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:54:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:54:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:54:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:54:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:54:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:54:21.272 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:54:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:54:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:54:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:54:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.273 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:54:21.274 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:54:21.274 INFO:teuthology.orchestra.run.smithi088.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:54:21.274 INFO:teuthology.orchestra.run.smithi088.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:54:21.274 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:54:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:54:21.275 INFO:teuthology.orchestra.run.smithi088.stdout: 2023-07-14T18:54:21.275 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2023-07-14T18:54:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:54:21.275 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2023-07-14T18:54:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:54:21.275 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.276 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2023-07-14T18:54:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:54:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:54:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:54:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.276 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.277 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.278 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.279 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:54:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:54:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:54:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:54:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:54:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:54:21.280 INFO:teuthology.orchestra.run.smithi102.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:54:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:54:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:54:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:54:21.281 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:54:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:54:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:54:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:54:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:54:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.282 INFO:teuthology.orchestra.run.smithi102.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:54:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:54:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:54:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:54:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:54:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:54:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:54:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:54:21.283 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:54:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:54:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:54:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:54:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:54:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:54:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:21.284 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:54:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:54:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:54:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:54:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:54:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.285 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:54:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:54:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:54:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:54:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:54:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:54:21.286 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:54:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:54:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:54:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:54:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:54:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:54:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:54:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:54:21.287 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:54:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:54:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:54:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:54:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:54:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:54:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:54:21.288 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:54:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:54:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:54:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:54:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:54:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:54:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:54:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:54:21.289 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:54:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:54:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:54:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:54:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:54:21.290 INFO:teuthology.orchestra.run.smithi102.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:54:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:54:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:54:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:54:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:54:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:54:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:54:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:54:21.291 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:54:21.292 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:54:21.292 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.292 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.292 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:21.292 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:54:21.292 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:54:21.292 INFO:teuthology.orchestra.run.smithi102.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:54:21.293 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:54:21.293 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:54:21.293 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:54:21.293 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:54:21.293 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2023-07-14T18:54:21.468 DEBUG:teuthology.parallel:result is None 2023-07-14T18:54:21.469 DEBUG:teuthology.parallel:result is None 2023-07-14T18:54:24.086 INFO:teuthology.orchestra.run.smithi078.stdout:Installed products updated. 2023-07-14T18:54:24.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:54:24.299 INFO:teuthology.orchestra.run.smithi078.stdout:Upgraded: 2023-07-14T18:54:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-07-14T18:54:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: librados2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: librbd1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.299 INFO:teuthology.orchestra.run.smithi078.stdout:Installed: 2023-07-14T18:54:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-07-14T18:54:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: automake-1.16.1-7.el8.noarch 2023-07-14T18:54:24.299 INFO:teuthology.orchestra.run.smithi078.stdout: bison-3.0.4-10.el8.x86_64 2023-07-14T18:54:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-grafana-dashboards-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-immutable-object-cache-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.300 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mds-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-diskprediction-local-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-rook-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mon-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.301 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-osd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-prometheus-alerts-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-radosgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-selinux-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-test-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-volume-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: cephadm-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.302 INFO:teuthology.orchestra.run.smithi078.stdout: cephfs-top-2:18.0.0-4940.g81c9387d.el8.noarch 2023-07-14T18:54:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-07-14T18:54:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: flex-2.6.1-9.el8.x86_64 2023-07-14T18:54:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: fuse-2.9.7-15.el8.x86_64 2023-07-14T18:54:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-07-14T18:54:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: jq-1.6-3.el8.x86_64 2023-07-14T18:54:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-07-14T18:54:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-07-14T18:54:24.303 INFO:teuthology.orchestra.run.smithi078.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-07-14T18:54:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: libcephsqlite-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-07-14T18:54:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:54:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: liboath-2.6.2-3.el8.x86_64 2023-07-14T18:54:24.304 INFO:teuthology.orchestra.run.smithi078.stdout: libpq-13.5-1.el8.x86_64 2023-07-14T18:54:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-07-14T18:54:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-07-14T18:54:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: librados-devel-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-07-14T18:54:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: librgw2-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-07-14T18:54:24.305 INFO:teuthology.orchestra.run.smithi078.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-07-14T18:54:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: libtool-2.4.6-25.el8.x86_64 2023-07-14T18:54:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: libunwind-1.3.1-3.el8.x86_64 2023-07-14T18:54:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-07-14T18:54:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-07-14T18:54:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-07-14T18:54:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-07-14T18:54:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-0.3.15-3.el8.x86_64 2023-07-14T18:54:24.306 INFO:teuthology.orchestra.run.smithi078.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-07-14T18:54:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-07-14T18:54:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-07-14T18:54:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-07-14T18:54:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-07-14T18:54:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-07-14T18:54:24.307 INFO:teuthology.orchestra.run.smithi078.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-07-14T18:54:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel-2.5.1-7.el8.noarch 2023-07-14T18:54:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-07-14T18:54:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-07-14T18:54:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-07-14T18:54:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-argparse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.308 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cephfs-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-07-14T18:54:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-07-14T18:54:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-07-14T18:54:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-07-14T18:54:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-07-14T18:54:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-07-14T18:54:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-6.2-6.el8.noarch 2023-07-14T18:54:24.309 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-07-14T18:54:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-07-14T18:54:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-07-14T18:54:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-07-14T18:54:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-07-14T18:54:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-07-14T18:54:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-07-14T18:54:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: python3-mako-1.0.6-13.el8.noarch 2023-07-14T18:54:24.310 INFO:teuthology.orchestra.run.smithi078.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-07-14T18:54:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-07-14T18:54:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-07-14T18:54:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-07-14T18:54:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-07-14T18:54:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-07-14T18:54:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-07-14T18:54:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: python3-portend-2.6-1.el8.noarch 2023-07-14T18:54:24.311 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-07-14T18:54:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-07-14T18:54:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-07-14T18:54:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-07-14T18:54:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pycparser-2.14-14.el8.noarch 2023-07-14T18:54:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pytz-2017.2-9.el8.noarch 2023-07-14T18:54:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rados-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.312 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-07-14T18:54:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-07-14T18:54:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rgw-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes-2.4.1-12.el8.noarch 2023-07-14T18:54:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa-4.8-1.el8.noarch 2023-07-14T18:54:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-07-14T18:54:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-07-14T18:54:24.313 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-07-14T18:54:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-07-14T18:54:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-07-14T18:54:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-07-14T18:54:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-07-14T18:54:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-07-14T18:54:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-07-14T18:54:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-07-14T18:54:24.314 INFO:teuthology.orchestra.run.smithi078.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-07-14T18:54:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-fuse-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-mirror-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: rbd-nbd-2:18.0.0-4940.g81c9387d.el8.x86_64 2023-07-14T18:54:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: socat-1.7.4.1-1.el8.x86_64 2023-07-14T18:54:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: thrift-0.13.0-2.el8.x86_64 2023-07-14T18:54:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: uuid-1.6.2-43.el8.x86_64 2023-07-14T18:54:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-07-14T18:54:24.315 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-07-14T18:54:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-07-14T18:54:24.316 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:54:24.316 INFO:teuthology.orchestra.run.smithi078.stdout:Complete! 2023-07-14T18:54:24.479 DEBUG:teuthology.parallel:result is None 2023-07-14T18:54:24.479 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:54:29.718 DEBUG:teuthology.orchestra.run.smithi078:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:54:29.746 INFO:teuthology.orchestra.run.smithi078.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:54:29.747 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:54:29.747 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:54:29.748 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:54:35.161 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:54:35.189 INFO:teuthology.orchestra.run.smithi088.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:54:35.190 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:54:35.190 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:54:35.191 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:54:38.452 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-07-14T18:54:38.484 INFO:teuthology.orchestra.run.smithi102.stdout:18.0.0-4940.g81c9387d.el8 2023-07-14T18:54:38.485 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-4940.g81c9387d.el8 2023-07-14T18:54:38.485 INFO:teuthology.task.install:The correct ceph version 18.0.0-4940.g81c9387d is installed. 2023-07-14T18:54:38.486 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-07-14T18:54:38.487 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:54:38.487 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:54:38.533 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:54:38.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:54:38.581 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:54:38.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-07-14T18:54:38.629 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-07-14T18:54:38.630 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:54:38.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:54:38.660 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:54:38.729 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:54:38.729 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:54:38.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:54:38.830 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:54:38.831 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2023-07-14T18:54:38.861 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-07-14T18:54:38.930 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-07-14T18:54:38.930 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:54:38.930 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:54:38.961 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:54:39.030 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:54:39.030 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:54:39.061 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:54:39.132 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:54:39.133 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2023-07-14T18:54:39.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-07-14T18:54:39.231 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-07-14T18:54:39.231 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:54:39.232 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:54:39.262 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:54:39.331 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:54:39.332 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:54:39.365 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:54:39.436 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:54:39.436 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2023-07-14T18:54:39.465 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-07-14T18:54:39.534 INFO:teuthology.run_tasks:Running task cephadm... 2023-07-14T18:54:39.629 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-14T18:54:39.630 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:54:39.641 INFO:tasks.cephadm:Cluster fsid is e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:54:39.642 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-07-14T18:54:39.642 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.78', 'mon.b': '172.21.15.88', 'mon.c': '172.21.15.102'} 2023-07-14T18:54:39.643 INFO:tasks.cephadm:First mon is mon.a on smithi078 2023-07-14T18:54:39.643 INFO:tasks.cephadm:First mgr is x 2023-07-14T18:54:39.643 INFO:tasks.cephadm:Normalizing hostnames... 2023-07-14T18:54:39.644 DEBUG:teuthology.orchestra.run.smithi078:> sudo hostname $(hostname -s) 2023-07-14T18:54:39.678 DEBUG:teuthology.orchestra.run.smithi088:> sudo hostname $(hostname -s) 2023-07-14T18:54:39.711 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2023-07-14T18:54:39.740 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-07-14T18:54:39.741 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:54:43.408 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '81c9387d2e7576da065747ec9c16d1c7d69016ec', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/71431/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-4940-g81c9387d', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-4940.g81c9387d'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-07-14 16:50:26.249583', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20230714.134335', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/flavors/default/', 'archs': ['source', 'x86_64', 'arm64'], 'distro': 'centos'}] 2023-07-14T18:54:43.716 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20230714.134335, sha1 81c9387d2e7576da065747ec9c16d1c7d69016ec from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=81c9387d2e7576da065747ec9c16d1c7d69016ec 2023-07-14T18:54:43.716 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm 2023-07-14T18:54:43.717 DEBUG:teuthology.orchestra.run.smithi078:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:44.581 INFO:teuthology.orchestra.run.smithi078.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:54 /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:44.581 DEBUG:teuthology.orchestra.run.smithi088:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:45.377 INFO:teuthology.orchestra.run.smithi088.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:54 /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:45.378 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20230714.134335/81c9387d2e7576da065747ec9c16d1c7d69016ec/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:46.164 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-rw-r--. 1 ubuntu ubuntu 670007 Jul 14 18:54 /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:46.165 DEBUG:teuthology.orchestra.run.smithi078:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:46.186 DEBUG:teuthology.orchestra.run.smithi088:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:46.207 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-07-14T18:54:46.242 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec on all hosts... 2023-07-14T18:54:46.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:54:46.245 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:54:46.249 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec pull 2023-07-14T18:54:47.934 INFO:teuthology.orchestra.run.smithi088.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:54:48.016 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:54:48.219 INFO:teuthology.orchestra.run.smithi078.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:55:23.941 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2023-07-14T18:55:23.942 INFO:teuthology.orchestra.run.smithi088.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:55:23.942 INFO:teuthology.orchestra.run.smithi088.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:55:23.942 INFO:teuthology.orchestra.run.smithi088.stdout: "repo_digests": [ 2023-07-14T18:55:23.942 INFO:teuthology.orchestra.run.smithi088.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:55:23.942 INFO:teuthology.orchestra.run.smithi088.stdout: ] 2023-07-14T18:55:23.943 INFO:teuthology.orchestra.run.smithi088.stdout:} 2023-07-14T18:55:25.007 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2023-07-14T18:55:25.007 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:55:25.007 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:55:25.007 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2023-07-14T18:55:25.007 INFO:teuthology.orchestra.run.smithi102.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:55:25.007 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2023-07-14T18:55:25.008 INFO:teuthology.orchestra.run.smithi102.stdout:} 2023-07-14T18:55:26.227 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T18:55:26.228 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T18:55:26.228 INFO:teuthology.orchestra.run.smithi078.stdout: "image_id": "219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705", 2023-07-14T18:55:26.229 INFO:teuthology.orchestra.run.smithi078.stdout: "repo_digests": [ 2023-07-14T18:55:26.229 INFO:teuthology.orchestra.run.smithi078.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:1302190083259ae87fcd27babf423c1dd04ae7417edb252031c1d861adeb3110" 2023-07-14T18:55:26.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2023-07-14T18:55:26.229 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T18:55:26.247 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph 2023-07-14T18:55:26.276 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph 2023-07-14T18:55:26.313 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2023-07-14T18:55:26.349 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /etc/ceph 2023-07-14T18:55:26.375 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /etc/ceph 2023-07-14T18:55:26.406 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2023-07-14T18:55:26.436 INFO:tasks.cephadm:Writing seed config... 2023-07-14T18:55:26.438 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-07-14T18:55:26.438 INFO:tasks.cephadm: override: [client] debug client = 20 2023-07-14T18:55:26.438 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-07-14T18:55:26.438 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-07-14T18:55:26.439 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-07-14T18:55:26.439 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-07-14T18:55:26.439 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-07-14T18:55:26.439 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-07-14T18:55:26.439 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2023-07-14T18:55:26.440 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2023-07-14T18:55:26.440 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2023-07-14T18:55:26.440 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2023-07-14T18:55:26.440 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-07-14T18:55:26.441 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-07-14T18:55:26.441 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-07-14T18:55:26.441 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-07-14T18:55:26.441 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-07-14T18:55:26.441 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-07-14T18:55:26.441 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-07-14T18:55:26.442 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2023-07-14T18:55:26.442 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-07-14T18:55:26.442 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-07-14T18:55:26.442 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-07-14T18:55:26.442 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-07-14T18:55:26.443 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-07-14T18:55:26.443 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-07-14T18:55:26.443 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-07-14T18:55:26.443 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-07-14T18:55:26.444 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2023-07-14T18:55:26.444 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2023-07-14T18:55:26.444 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-07-14T18:55:26.444 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-07-14T18:55:26.444 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2023-07-14T18:55:26.445 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-07-14T18:55:26.445 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-07-14T18:55:26.445 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-07-14T18:55:26.445 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-07-14T18:55:26.445 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-07-14T18:55:26.446 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-07-14T18:55:26.446 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2023-07-14T18:55:26.446 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2023-07-14T18:55:26.447 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:55:26.447 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-07-14T18:55:26.464 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e2ca1a0e-2277-11ee-9b33-001a4aab830c [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-07-14T18:55:26.464 DEBUG:teuthology.orchestra.run.smithi078:mon.a> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.a.service 2023-07-14T18:55:26.506 DEBUG:teuthology.orchestra.run.smithi078:mgr.x> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.x.service 2023-07-14T18:55:26.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T18:55:26.532 INFO:tasks.cephadm:Bootstrapping... 2023-07-14T18:55:26.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec -v bootstrap --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.78 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:55:26.557 INFO:journalctl@ceph.mgr.x.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T18:55:26.688 INFO:teuthology.orchestra.run.smithi078.stdout:-------------------------------------------------------------------------------- 2023-07-14T18:55:26.689 INFO:teuthology.orchestra.run.smithi078.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec', '-v', 'bootstrap', '--fsid', 'e2ca1a0e-2277-11ee-9b33-001a4aab830c', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.78', '--skip-admin-label'] 2023-07-14T18:55:26.832 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:55:26.833 INFO:teuthology.orchestra.run.smithi078.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-07-14T18:55:26.833 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying podman|docker is present... 2023-07-14T18:55:26.974 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:55:26.975 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying lvm2 is present... 2023-07-14T18:55:26.975 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying time synchronization is in place... 2023-07-14T18:55:26.982 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T18:55:26.982 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T18:55:26.987 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T18:55:26.988 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-07-14T18:55:26.993 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2023-07-14T18:55:26.999 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2023-07-14T18:55:26.999 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2023-07-14T18:55:26.999 INFO:teuthology.orchestra.run.smithi078.stdout:Repeating the final host check... 2023-07-14T18:55:27.140 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 4.1.1 2023-07-14T18:55:27.141 INFO:teuthology.orchestra.run.smithi078.stdout:podman (/bin/podman) version 4.1.1 is present 2023-07-14T18:55:27.141 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl is present 2023-07-14T18:55:27.141 INFO:teuthology.orchestra.run.smithi078.stdout:lvcreate is present 2023-07-14T18:55:27.146 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-07-14T18:55:27.147 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-07-14T18:55:27.152 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-07-14T18:55:27.152 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-07-14T18:55:27.158 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout enabled 2023-07-14T18:55:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout active 2023-07-14T18:55:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:Unit chronyd.service is enabled and running 2023-07-14T18:55:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:Host looks OK 2023-07-14T18:55:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:Cluster fsid: e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:55:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:Acquiring lock 139803527662056 on /run/cephadm/e2ca1a0e-2277-11ee-9b33-001a4aab830c.lock 2023-07-14T18:55:27.164 INFO:teuthology.orchestra.run.smithi078.stdout:Lock 139803527662056 acquired on /run/cephadm/e2ca1a0e-2277-11ee-9b33-001a4aab830c.lock 2023-07-14T18:55:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 3300 ... 2023-07-14T18:55:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying IP 172.21.15.78 port 6789 ... 2023-07-14T18:55:27.165 INFO:teuthology.orchestra.run.smithi078.stdout:Base mon IP(s) is [172.21.15.78:3300, 172.21.15.78:6789], mon addrv is [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2023-07-14T18:55:27.168 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2023-07-14T18:55:27.168 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.78 metric 100 2023-07-14T18:55:27.171 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-07-14T18:55:27.171 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2023-07-14T18:55:27.171 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2023-07-14T18:55:27.173 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-07-14T18:55:27.173 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-07-14T18:55:27.173 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T18:55:27.173 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-07-14T18:55:27.174 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout inet6 fe80::8e6d:82b2:36a8:5b8d/64 scope link noprefixroute 2023-07-14T18:55:27.174 INFO:teuthology.orchestra.run.smithi078.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-07-14T18:55:27.174 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2023-07-14T18:55:27.174 INFO:teuthology.orchestra.run.smithi078.stdout:Mon IP `172.21.15.78` is in CIDR network `172.21.0.0/20` 2023-07-14T18:55:27.175 INFO:teuthology.orchestra.run.smithi078.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-07-14T18:55:27.175 INFO:teuthology.orchestra.run.smithi078.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-07-14T18:55:27.175 INFO:teuthology.orchestra.run.smithi078.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:55:27.823 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stdout 219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T18:55:27.823 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec... 2023-07-14T18:55:27.823 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Getting image source signatures 2023-07-14T18:55:27.823 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:0e454b422ba90e2ad13e636c40610401086a3576fe973b0460943839fe5f2508 2023-07-14T18:55:27.824 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying blob sha256:6022c611d474f2b306ddd1edfc8360123ece5266119096f25fde843d7967e636 2023-07-14T18:55:27.824 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Copying config sha256:219b180907a819a02c0d1790ec69e152120a3974adf061aaf947f90f6ccf9705 2023-07-14T18:55:27.824 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Writing manifest to image destination 2023-07-14T18:55:27.824 INFO:teuthology.orchestra.run.smithi078.stdout:/bin/podman: stderr Storing signatures 2023-07-14T18:55:28.678 INFO:teuthology.orchestra.run.smithi078.stdout:ceph: stdout ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T18:55:28.679 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph version: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev) 2023-07-14T18:55:28.679 INFO:teuthology.orchestra.run.smithi078.stdout:Extracting ceph user uid/gid from container image... 2023-07-14T18:55:29.423 INFO:teuthology.orchestra.run.smithi078.stdout:stat: stdout 167 167 2023-07-14T18:55:29.424 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial keys... 2023-07-14T18:55:30.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQAhmrFkxCHHLhAAzTLIMdDh8WOJEARJAIUiHQ== 2023-07-14T18:55:30.915 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQAimrFkNGsKIRAAv6AdMJDlbWgyGLIwUIIKGQ== 2023-07-14T18:55:31.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph-authtool: stdout AQAjmrFkSeKeExAAgwv9fJ0TkvBMd2fkaCysGg== 2023-07-14T18:55:31.662 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial monmap... 2023-07-14T18:55:32.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T18:55:32.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-07-14T18:55:32.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:55:32.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T18:55:32.422 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool for a [v2:172.21.15.78:3300,v1:172.21.15.78:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-07-14T18:55:32.423 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = pacific 2023-07-14T18:55:32.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: set fsid to e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:55:32.423 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-07-14T18:55:32.423 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:55:32.423 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mon... 2023-07-14T18:55:34.577 INFO:teuthology.orchestra.run.smithi078.stdout:create mon.a on 2023-07-14T18:55:34.890 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-07-14T18:55:34.979 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-07-14T18:55:35.390 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c.target. 2023-07-14T18:55:35.391 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c.target → /etc/systemd/system/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c.target. 2023-07-14T18:55:35.643 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.a 2023-07-14T18:55:35.643 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.a.service: Unit ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.a.service not loaded. 2023-07-14T18:55:35.728 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c.target.wants/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@.service. 2023-07-14T18:55:35.995 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:35 smithi078 systemd[1]: Starting Ceph mon.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T18:55:36.231 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:55:36.231 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2023-07-14T18:55:36.238 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:55:36.239 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-07-14T18:55:36.239 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2023-07-14T18:55:36.239 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T18:55:36.239 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon to start... 2023-07-14T18:55:36.240 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mon... 2023-07-14T18:55:36.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:35 smithi078 podman[126881]: 2023-07-14T18:55:36.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:36 smithi078 bash[126881]: 23d1aa4feae8095577c7702d7f8a66d4093a90115621f392e729d61c21c3abc0 2023-07-14T18:55:36.248 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:36 smithi078 systemd[1]: Started Ceph mon.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T18:55:36.520 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:36 smithi078 ceph-mon[126921]: mkfs e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:55:36.521 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:36 smithi078 ceph-mon[126921]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:55:37.191 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout cluster: 2023-07-14T18:55:37.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout id: e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:55:37.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-07-14T18:55:37.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:37.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout services: 2023-07-14T18:55:37.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.542737s) 2023-07-14T18:55:37.192 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-07-14T18:55:37.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-07-14T18:55:37.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:37.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout data: 2023-07-14T18:55:37.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-07-14T18:55:37.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-07-14T18:55:37.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-07-14T18:55:37.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout pgs: 2023-07-14T18:55:37.193 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:37.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.705+0000 7f1f1ea7d700 1 Processor -- start 2023-07-14T18:55:37.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.705+0000 7f1f1ea7d700 1 -- start start 2023-07-14T18:55:37.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.705+0000 7f1f1ea7d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18103890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:37.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.705+0000 7f1f1ea7d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18103dd0 con 0x7f1f18103480 2023-07-14T18:55:37.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.705+0000 7f1f1c819700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:37.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.705+0000 7f1f1c819700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33120/0 (socket says 172.21.15.78:33120) 2023-07-14T18:55:37.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.705+0000 7f1f1c819700 1 -- 172.21.15.78:0/1839037396 learned_addr learned my addr 172.21.15.78:0/1839037396 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:37.194 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.706+0000 7f1f1c819700 1 -- 172.21.15.78:0/1839037396 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f18103f10 con 0x7f1f18103480 2023-07-14T18:55:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.706+0000 7f1f1c819700 1 --2- 172.21.15.78:0/1839037396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18103890 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0098a0 tx=0x7f1f0c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbd002105b558db server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.706+0000 7f1f177fe700 1 -- 172.21.15.78:0/1839037396 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f0c004160 con 0x7f1f18103480 2023-07-14T18:55:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.706+0000 7f1f177fe700 1 -- 172.21.15.78:0/1839037396 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c0042c0 con 0x7f1f18103480 2023-07-14T18:55:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.706+0000 7f1f177fe700 1 -- 172.21.15.78:0/1839037396 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f0c0045a0 con 0x7f1f18103480 2023-07-14T18:55:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.706+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/1839037396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 msgr2=0x7f1f18103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 --2- 172.21.15.78:0/1839037396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18103890 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c0098a0 tx=0x7f1f0c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:55:37.195 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/1839037396 shutdown_connections 2023-07-14T18:55:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 --2- 172.21.15.78:0/1839037396 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18103890 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/1839037396 >> 172.21.15.78:0/1839037396 conn(0x7f1f180fee80 msgr2=0x7f1f18101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/1839037396 shutdown_connections 2023-07-14T18:55:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/1839037396 wait complete. 2023-07-14T18:55:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 Processor -- start 2023-07-14T18:55:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 -- start start 2023-07-14T18:55:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18105e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:37.196 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1ea7d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f18106350 con 0x7f1f18103480 2023-07-14T18:55:37.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1c819700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18105e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:37.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1c819700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18105e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33122/0 (socket says 172.21.15.78:33122) 2023-07-14T18:55:37.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.707+0000 7f1f1c819700 1 -- 172.21.15.78:0/3435250098 learned_addr learned my addr 172.21.15.78:0/3435250098 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:37.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f1c819700 1 -- 172.21.15.78:0/3435250098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f0c009550 con 0x7f1f18103480 2023-07-14T18:55:37.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f1c819700 1 --2- 172.21.15.78:0/3435250098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18105e10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c004000 tx=0x7f1f0c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:37.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f15ffb700 1 -- 172.21.15.78:0/3435250098 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f0c011630 con 0x7f1f18103480 2023-07-14T18:55:37.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f15ffb700 1 -- 172.21.15.78:0/3435250098 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f0c011790 con 0x7f1f18103480 2023-07-14T18:55:37.197 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f18104460 con 0x7f1f18103480 2023-07-14T18:55:37.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f15ffb700 1 -- 172.21.15.78:0/3435250098 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1f0c011a30 con 0x7f1f18103480 2023-07-14T18:55:37.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f18104880 con 0x7f1f18103480 2023-07-14T18:55:37.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f15ffb700 1 -- 172.21.15.78:0/3435250098 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1f0c022070 con 0x7f1f18103480 2023-07-14T18:55:37.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.708+0000 7f1f15ffb700 1 -- 172.21.15.78:0/3435250098 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1f0c01c450 con 0x7f1f18103480 2023-07-14T18:55:37.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.709+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f18105290 con 0x7f1f18103480 2023-07-14T18:55:37.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.711+0000 7f1f15ffb700 1 -- 172.21.15.78:0/3435250098 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f1f0c01b450 con 0x7f1f18103480 2023-07-14T18:55:37.198 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.780+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f1f181919d0 con 0x7f1f18103480 2023-07-14T18:55:37.199 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.780+0000 7f1f15ffb700 1 -- 172.21.15.78:0/3435250098 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f1f0c01b5f0 con 0x7f1f18103480 2023-07-14T18:55:37.199 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.781+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 msgr2=0x7f1f18105e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:37.199 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.781+0000 7f1f1ea7d700 1 --2- 172.21.15.78:0/3435250098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18105e10 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1f0c004000 tx=0x7f1f0c004750 comp rx=0 tx=0).stop 2023-07-14T18:55:37.199 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.781+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 shutdown_connections 2023-07-14T18:55:37.199 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.781+0000 7f1f1ea7d700 1 --2- 172.21.15.78:0/3435250098 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1f18103480 0x7f1f18105e10 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:37.199 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.781+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 >> 172.21.15.78:0/3435250098 conn(0x7f1f180fee80 msgr2=0x7f1f1818cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:37.199 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.781+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 shutdown_connections 2023-07-14T18:55:37.199 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:36.781+0000 7f1f1ea7d700 1 -- 172.21.15.78:0/3435250098 wait complete. 2023-07-14T18:55:37.200 INFO:teuthology.orchestra.run.smithi078.stdout:mon is available 2023-07-14T18:55:37.200 INFO:teuthology.orchestra.run.smithi078.stdout:Assimilating anything we can from ceph.conf... 2023-07-14T18:55:37.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:37 smithi078 ceph-mon[126921]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:55:37.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:37 smithi078 ceph-mon[126921]: monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} removed_ranks: {} 2023-07-14T18:55:37.417 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:37 smithi078 ceph-mon[126921]: fsmap 2023-07-14T18:55:37.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:37 smithi078 ceph-mon[126921]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T18:55:37.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:37 smithi078 ceph-mon[126921]: mgrmap e1: no daemons active 2023-07-14T18:55:37.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:37 smithi078 ceph-mon[126921]: from='client.? 172.21.15.78:0/3435250098' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-07-14T18:55:38.136 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:38.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T18:55:38.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:55:38.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.78:3300,v1:172.21.15.78:6789] 2023-07-14T18:55:38.137 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T18:55:38.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T18:55:38.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T18:55:38.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T18:55:38.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:38.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T18:55:38.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T18:55:38.138 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:38.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T18:55:38.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T18:55:38.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T18:55:38.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T18:55:38.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T18:55:38.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T18:55:38.139 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.682+0000 7f136cadf700 1 Processor -- start 2023-07-14T18:55:38.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.682+0000 7f136cadf700 1 -- start start 2023-07-14T18:55:38.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136cadf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f1368103c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:38.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136cadf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1368104160 con 0x7f1368103810 2023-07-14T18:55:38.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f1368103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:38.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f1368103c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33124/0 (socket says 172.21.15.78:33124) 2023-07-14T18:55:38.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136659c700 1 -- 172.21.15.78:0/793898534 learned_addr learned my addr 172.21.15.78:0/793898534 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:38.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136659c700 1 -- 172.21.15.78:0/793898534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13681042a0 con 0x7f1368103810 2023-07-14T18:55:38.140 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136659c700 1 --2- 172.21.15.78:0/793898534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f1368103c20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f135c00aa40 tx=0x7f135c00ad50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bf39b50bd4f1d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:38.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136559a700 1 -- 172.21.15.78:0/793898534 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f135c004160 con 0x7f1368103810 2023-07-14T18:55:38.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136559a700 1 -- 172.21.15.78:0/793898534 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c0042c0 con 0x7f1368103810 2023-07-14T18:55:38.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.683+0000 7f136559a700 1 -- 172.21.15.78:0/793898534 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f135c004580 con 0x7f1368103810 2023-07-14T18:55:38.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.684+0000 7f136cadf700 1 -- 172.21.15.78:0/793898534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 msgr2=0x7f1368103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:38.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.684+0000 7f136cadf700 1 --2- 172.21.15.78:0/793898534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f1368103c20 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f135c00aa40 tx=0x7f135c00ad50 comp rx=0 tx=0).stop 2023-07-14T18:55:38.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.684+0000 7f136cadf700 1 -- 172.21.15.78:0/793898534 shutdown_connections 2023-07-14T18:55:38.141 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.684+0000 7f136cadf700 1 --2- 172.21.15.78:0/793898534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f1368103c20 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:38.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.684+0000 7f136cadf700 1 -- 172.21.15.78:0/793898534 >> 172.21.15.78:0/793898534 conn(0x7f13680fee80 msgr2=0x7f1368101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:38.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.684+0000 7f136cadf700 1 -- 172.21.15.78:0/793898534 shutdown_connections 2023-07-14T18:55:38.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.684+0000 7f136cadf700 1 -- 172.21.15.78:0/793898534 wait complete. 2023-07-14T18:55:38.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136cadf700 1 Processor -- start 2023-07-14T18:55:38.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136cadf700 1 -- start start 2023-07-14T18:55:38.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136cadf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f136818c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:38.142 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136cadf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f136818ca90 con 0x7f1368103810 2023-07-14T18:55:38.143 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f136818c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:38.143 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f136818c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33126/0 (socket says 172.21.15.78:33126) 2023-07-14T18:55:38.143 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136659c700 1 -- 172.21.15.78:0/4283714561 learned_addr learned my addr 172.21.15.78:0/4283714561 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:38.143 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136659c700 1 -- 172.21.15.78:0/4283714561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c00a6f0 con 0x7f1368103810 2023-07-14T18:55:38.143 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.685+0000 7f136659c700 1 --2- 172.21.15.78:0/4283714561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f136818c550 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f135c00afd0 tx=0x7f135c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:38.144 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.686+0000 7f13577fe700 1 -- 172.21.15.78:0/4283714561 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f135c01d070 con 0x7f1368103810 2023-07-14T18:55:38.144 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.686+0000 7f13577fe700 1 -- 172.21.15.78:0/4283714561 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135c003aa0 con 0x7f1368103810 2023-07-14T18:55:38.144 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.686+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f136818aba0 con 0x7f1368103810 2023-07-14T18:55:38.144 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.686+0000 7f13577fe700 1 -- 172.21.15.78:0/4283714561 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f135c003d80 con 0x7f1368103810 2023-07-14T18:55:38.144 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.686+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f136818afc0 con 0x7f1368103810 2023-07-14T18:55:38.144 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.686+0000 7f13577fe700 1 -- 172.21.15.78:0/4283714561 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f135c02b440 con 0x7f1368103810 2023-07-14T18:55:38.144 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.686+0000 7f13577fe700 1 -- 172.21.15.78:0/4283714561 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f135c021460 con 0x7f1368103810 2023-07-14T18:55:38.144 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.687+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f136818b9d0 con 0x7f1368103810 2023-07-14T18:55:38.145 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.689+0000 7f13577fe700 1 -- 172.21.15.78:0/4283714561 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f135c026070 con 0x7f1368103810 2023-07-14T18:55:38.145 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.755+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1368191d30 con 0x7f1368103810 2023-07-14T18:55:38.145 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.759+0000 7f13577fe700 1 -- 172.21.15.78:0/4283714561 <== mon.0 v2:172.21.15.78:3300/0 7 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f135c021ae0 con 0x7f1368103810 2023-07-14T18:55:38.145 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.759+0000 7f13577fe700 1 -- 172.21.15.78:0/4283714561 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f1368191d30 con 0x7f1368103810 2023-07-14T18:55:38.145 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.760+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 msgr2=0x7f136818c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:38.145 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.760+0000 7f136cadf700 1 --2- 172.21.15.78:0/4283714561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f136818c550 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f135c00afd0 tx=0x7f135c004750 comp rx=0 tx=0).stop 2023-07-14T18:55:38.145 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.760+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 shutdown_connections 2023-07-14T18:55:38.146 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.760+0000 7f136cadf700 1 --2- 172.21.15.78:0/4283714561 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1368103810 0x7f136818c550 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:38.146 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.760+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 >> 172.21.15.78:0/4283714561 conn(0x7f13680fee80 msgr2=0x7f136818f490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:38.146 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.760+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 shutdown_connections 2023-07-14T18:55:38.146 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:37.760+0000 7f136cadf700 1 -- 172.21.15.78:0/4283714561 wait complete. 2023-07-14T18:55:38.146 INFO:teuthology.orchestra.run.smithi078.stdout:Generating new minimal ceph.conf... 2023-07-14T18:55:38.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:38 smithi078 ceph-mon[126921]: from='client.? 172.21.15.78:0/4283714561' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T18:55:38.412 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:38 smithi078 ceph-mon[126921]: from='client.? 172.21.15.78:0/4283714561' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-07-14T18:55:39.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.623+0000 7f237f6c3700 1 Processor -- start 2023-07-14T18:55:39.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.623+0000 7f237f6c3700 1 -- start start 2023-07-14T18:55:39.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.623+0000 7f237f6c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f2378104490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:39.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.623+0000 7f237f6c3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23781049d0 con 0x7f2378104080 2023-07-14T18:55:39.079 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.623+0000 7f237d45f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f2378104490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:39.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.624+0000 7f237d45f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f2378104490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33128/0 (socket says 172.21.15.78:33128) 2023-07-14T18:55:39.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.624+0000 7f237d45f700 1 -- 172.21.15.78:0/1030765587 learned_addr learned my addr 172.21.15.78:0/1030765587 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:39.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.624+0000 7f237d45f700 1 -- 172.21.15.78:0/1030765587 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2378104b10 con 0x7f2378104080 2023-07-14T18:55:39.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.624+0000 7f237d45f700 1 --2- 172.21.15.78:0/1030765587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f2378104490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f236800bb40 tx=0x7f236800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d79efd46af93f930 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:39.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.624+0000 7f236ffff700 1 -- 172.21.15.78:0/1030765587 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f236800dbe0 con 0x7f2378104080 2023-07-14T18:55:39.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.624+0000 7f236ffff700 1 -- 172.21.15.78:0/1030765587 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f23680044a0 con 0x7f2378104080 2023-07-14T18:55:39.080 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.624+0000 7f236ffff700 1 -- 172.21.15.78:0/1030765587 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2368003da0 con 0x7f2378104080 2023-07-14T18:55:39.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 -- 172.21.15.78:0/1030765587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 msgr2=0x7f2378104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:39.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 --2- 172.21.15.78:0/1030765587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f2378104490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f236800bb40 tx=0x7f236800be50 comp rx=0 tx=0).stop 2023-07-14T18:55:39.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 -- 172.21.15.78:0/1030765587 shutdown_connections 2023-07-14T18:55:39.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 --2- 172.21.15.78:0/1030765587 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f2378104490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:39.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 -- 172.21.15.78:0/1030765587 >> 172.21.15.78:0/1030765587 conn(0x7f23780ff710 msgr2=0x7f2378101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:39.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 -- 172.21.15.78:0/1030765587 shutdown_connections 2023-07-14T18:55:39.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 -- 172.21.15.78:0/1030765587 wait complete. 2023-07-14T18:55:39.081 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 Processor -- start 2023-07-14T18:55:39.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 -- start start 2023-07-14T18:55:39.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f237819af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:39.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.625+0000 7f237f6c3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f237819b4d0 con 0x7f2378104080 2023-07-14T18:55:39.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f237d45f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f237819af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:39.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f237d45f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f237819af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33130/0 (socket says 172.21.15.78:33130) 2023-07-14T18:55:39.082 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f237d45f700 1 -- 172.21.15.78:0/442105876 learned_addr learned my addr 172.21.15.78:0/442105876 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f237d45f700 1 -- 172.21.15.78:0/442105876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f236800b7f0 con 0x7f2378104080 2023-07-14T18:55:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f237d45f700 1 --2- 172.21.15.78:0/442105876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f237819af90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f236800e010 tx=0x7f2368003f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f236e7fc700 1 -- 172.21.15.78:0/442105876 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2368010630 con 0x7f2378104080 2023-07-14T18:55:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f236e7fc700 1 -- 172.21.15.78:0/442105876 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2368010c00 con 0x7f2378104080 2023-07-14T18:55:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f237819b6d0 con 0x7f2378104080 2023-07-14T18:55:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f236e7fc700 1 -- 172.21.15.78:0/442105876 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2368021c00 con 0x7f2378104080 2023-07-14T18:55:39.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f237819ba90 con 0x7f2378104080 2023-07-14T18:55:39.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f236e7fc700 1 -- 172.21.15.78:0/442105876 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f236802a6f0 con 0x7f2378104080 2023-07-14T18:55:39.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.626+0000 7f236e7fc700 1 -- 172.21.15.78:0/442105876 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2368019750 con 0x7f2378104080 2023-07-14T18:55:39.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.627+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23781909b0 con 0x7f2378104080 2023-07-14T18:55:39.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.629+0000 7f236e7fc700 1 -- 172.21.15.78:0/442105876 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f236802f060 con 0x7f2378104080 2023-07-14T18:55:39.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.696+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f2378106020 con 0x7f2378104080 2023-07-14T18:55:39.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.696+0000 7f236e7fc700 1 -- 172.21.15.78:0/442105876 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f2368021410 con 0x7f2378104080 2023-07-14T18:55:39.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.697+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 msgr2=0x7f237819af90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:39.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.697+0000 7f237f6c3700 1 --2- 172.21.15.78:0/442105876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f237819af90 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f236800e010 tx=0x7f2368003f70 comp rx=0 tx=0).stop 2023-07-14T18:55:39.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.697+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 shutdown_connections 2023-07-14T18:55:39.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.697+0000 7f237f6c3700 1 --2- 172.21.15.78:0/442105876 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2378104080 0x7f237819af90 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:39.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.697+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 >> 172.21.15.78:0/442105876 conn(0x7f23780ff710 msgr2=0x7f237818daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:39.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.697+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 shutdown_connections 2023-07-14T18:55:39.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:38.697+0000 7f237f6c3700 1 -- 172.21.15.78:0/442105876 wait complete. 2023-07-14T18:55:39.085 INFO:teuthology.orchestra.run.smithi078.stdout:Restarting the monitor... 2023-07-14T18:55:39.400 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:39 smithi078 systemd[1]: Stopping Ceph mon.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T18:55:39.401 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:39 smithi078 ceph-mon[126921]: from='client.? 172.21.15.78:0/442105876' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:55:39.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:39 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-a[126897]: 2023-07-14T18:55:39.394+0000 7f7bf8a95700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T18:55:39.661 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:39 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-a[126897]: 2023-07-14T18:55:39.394+0000 7f7bf8a95700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-07-14T18:55:40.006 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:39 smithi078 bash[127253]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-a 2023-07-14T18:55:40.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 systemd[1]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.a.service: Succeeded. 2023-07-14T18:55:40.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 systemd[1]: Stopped Ceph mon.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T18:55:40.293 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 systemd[1]: Starting Ceph mon.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T18:55:40.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 podman[127385]: 2023-07-14T18:55:40.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T18:55:40.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T18:55:40.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: pidfile_write: ignore empty --pid-file 2023-07-14T18:55:40.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: load: jerasure load: lrc 2023-07-14T18:55:40.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: RocksDB version: 7.9.2 2023-07-14T18:55:40.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Git sha 0 2023-07-14T18:55:40.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T18:55:40.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: DB SUMMARY 2023-07-14T18:55:40.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: DB Session ID: CE3AGZHGL772HQ2MNG4H 2023-07-14T18:55:40.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: CURRENT file: CURRENT 2023-07-14T18:55:40.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T18:55:40.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-07-14T18:55:40.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-07-14T18:55:40.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97152 ; 2023-07-14T18:55:40.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.error_if_exists: 0 2023-07-14T18:55:40.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.create_if_missing: 0 2023-07-14T18:55:40.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.paranoid_checks: 1 2023-07-14T18:55:40.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T18:55:40.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T18:55:40.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T18:55:40.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.env: 0x5630d5144c60 2023-07-14T18:55:40.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T18:55:40.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.info_log: 0x5630d73572a0 2023-07-14T18:55:40.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T18:55:40.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.statistics: (nil) 2023-07-14T18:55:40.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.use_fsync: 0 2023-07-14T18:55:40.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_log_file_size: 0 2023-07-14T18:55:40.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T18:55:40.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T18:55:40.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T18:55:40.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T18:55:40.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.allow_fallocate: 1 2023-07-14T18:55:40.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T18:55:40.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T18:55:40.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.use_direct_reads: 0 2023-07-14T18:55:40.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T18:55:40.554 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T18:55:40.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.db_log_dir: 2023-07-14T18:55:40.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.wal_dir: 2023-07-14T18:55:40.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T18:55:40.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T18:55:40.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T18:55:40.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T18:55:40.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T18:55:40.555 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T18:55:40.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T18:55:40.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T18:55:40.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.write_buffer_manager: 0x5630d72783c0 2023-07-14T18:55:40.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T18:55:40.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T18:55:40.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T18:55:40.556 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T18:55:40.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T18:55:40.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T18:55:40.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T18:55:40.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T18:55:40.557 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.unordered_write: 0 2023-07-14T18:55:40.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T18:55:40.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T18:55:40.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T18:55:40.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T18:55:40.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.row_cache: None 2023-07-14T18:55:40.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.wal_filter: None 2023-07-14T18:55:40.558 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T18:55:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T18:55:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.two_write_queues: 0 2023-07-14T18:55:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T18:55:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.wal_compression: 0 2023-07-14T18:55:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.atomic_flush: 0 2023-07-14T18:55:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T18:55:40.559 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T18:55:40.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T18:55:40.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.log_readahead_size: 0 2023-07-14T18:55:40.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T18:55:40.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T18:55:40.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T18:55:40.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T18:55:40.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T18:55:40.560 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T18:55:40.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T18:55:40.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_background_jobs: 2 2023-07-14T18:55:40.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_background_compactions: -1 2023-07-14T18:55:40.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_subcompactions: 1 2023-07-14T18:55:40.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T18:55:40.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T18:55:40.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T18:55:40.561 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T18:55:40.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T18:55:40.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T18:55:40.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T18:55:40.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T18:55:40.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_open_files: -1 2023-07-14T18:55:40.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T18:55:40.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T18:55:40.562 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T18:55:40.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T18:55:40.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_background_flushes: -1 2023-07-14T18:55:40.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Compression algorithms supported: 2023-07-14T18:55:40.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T18:55:40.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: kZSTD supported: 0 2023-07-14T18:55:40.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: kXpressCompression supported: 0 2023-07-14T18:55:40.563 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T18:55:40.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: kZlibCompression supported: 1 2023-07-14T18:55:40.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: kSnappyCompression supported: 1 2023-07-14T18:55:40.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: kLZ4Compression supported: 1 2023-07-14T18:55:40.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: kBZip2Compression supported: 0 2023-07-14T18:55:40.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T18:55:40.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T18:55:40.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-07-14T18:55:40.564 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T18:55:40.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T18:55:40.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.merge_operator: 2023-07-14T18:55:40.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_filter: None 2023-07-14T18:55:40.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T18:55:40.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T18:55:40.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T18:55:40.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T18:55:40.565 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5630d73573a0) 2023-07-14T18:55:40.566 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks: 1 2023-07-14T18:55:40.566 INFO:journalctl@ceph.mon.a.smithi078.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T18:55:40.566 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T18:55:40.566 INFO:journalctl@ceph.mon.a.smithi078.stdout: pin_top_level_index_and_filter: 1 2023-07-14T18:55:40.566 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_type: 0 2023-07-14T18:55:40.566 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_index_type: 0 2023-07-14T18:55:40.566 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_shortening: 1 2023-07-14T18:55:40.566 INFO:journalctl@ceph.mon.a.smithi078.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T18:55:40.567 INFO:journalctl@ceph.mon.a.smithi078.stdout: checksum: 4 2023-07-14T18:55:40.567 INFO:journalctl@ceph.mon.a.smithi078.stdout: no_block_cache: 0 2023-07-14T18:55:40.567 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache: 0x5630d73171f0 2023-07-14T18:55:40.567 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_name: BinnedLRUCache 2023-07-14T18:55:40.567 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_options: 2023-07-14T18:55:40.567 INFO:journalctl@ceph.mon.a.smithi078.stdout: capacity : 536870912 2023-07-14T18:55:40.567 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_shard_bits : 4 2023-07-14T18:55:40.567 INFO:journalctl@ceph.mon.a.smithi078.stdout: strict_capacity_limit : 0 2023-07-14T18:55:40.568 INFO:journalctl@ceph.mon.a.smithi078.stdout: high_pri_pool_ratio: 0.000 2023-07-14T18:55:40.568 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_cache_compressed: (nil) 2023-07-14T18:55:40.568 INFO:journalctl@ceph.mon.a.smithi078.stdout: persistent_cache: (nil) 2023-07-14T18:55:40.568 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size: 4096 2023-07-14T18:55:40.568 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_size_deviation: 10 2023-07-14T18:55:40.568 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_restart_interval: 16 2023-07-14T18:55:40.568 INFO:journalctl@ceph.mon.a.smithi078.stdout: index_block_restart_interval: 1 2023-07-14T18:55:40.568 INFO:journalctl@ceph.mon.a.smithi078.stdout: metadata_block_size: 4096 2023-07-14T18:55:40.569 INFO:journalctl@ceph.mon.a.smithi078.stdout: partition_filters: 0 2023-07-14T18:55:40.569 INFO:journalctl@ceph.mon.a.smithi078.stdout: use_delta_encoding: 1 2023-07-14T18:55:40.569 INFO:journalctl@ceph.mon.a.smithi078.stdout: filter_policy: bloomfilter 2023-07-14T18:55:40.569 INFO:journalctl@ceph.mon.a.smithi078.stdout: whole_key_filtering: 1 2023-07-14T18:55:40.569 INFO:journalctl@ceph.mon.a.smithi078.stdout: verify_compression: 0 2023-07-14T18:55:40.569 INFO:journalctl@ceph.mon.a.smithi078.stdout: read_amp_bytes_per_bit: 0 2023-07-14T18:55:40.569 INFO:journalctl@ceph.mon.a.smithi078.stdout: format_version: 5 2023-07-14T18:55:40.569 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_index_compression: 1 2023-07-14T18:55:40.570 INFO:journalctl@ceph.mon.a.smithi078.stdout: block_align: 0 2023-07-14T18:55:40.570 INFO:journalctl@ceph.mon.a.smithi078.stdout: max_auto_readahead_size: 262144 2023-07-14T18:55:40.570 INFO:journalctl@ceph.mon.a.smithi078.stdout: prepopulate_block_cache: 0 2023-07-14T18:55:40.570 INFO:journalctl@ceph.mon.a.smithi078.stdout: initial_auto_readahead_size: 8192 2023-07-14T18:55:40.570 INFO:journalctl@ceph.mon.a.smithi078.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T18:55:40.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T18:55:40.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T18:55:40.570 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression: NoCompression 2023-07-14T18:55:40.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T18:55:40.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T18:55:40.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T18:55:40.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.num_levels: 7 2023-07-14T18:55:40.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T18:55:40.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T18:55:40.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T18:55:40.571 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T18:55:40.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T18:55:40.572 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T18:55:40.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T18:55:40.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:55:40.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T18:55:40.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T18:55:40.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:55:40.573 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:55:40.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T18:55:40.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T18:55:40.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T18:55:40.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T18:55:40.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:55:40.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:55:40.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T18:55:40.574 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T18:55:40.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:55:40.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T18:55:40.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T18:55:40.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T18:55:40.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T18:55:40.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T18:55:40.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T18:55:40.575 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T18:55:40.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T18:55:40.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T18:55:40.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T18:55:40.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T18:55:40.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T18:55:40.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T18:55:40.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T18:55:40.576 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T18:55:40.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T18:55:40.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T18:55:40.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T18:55:40.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T18:55:40.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T18:55:40.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T18:55:40.577 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T18:55:40.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T18:55:40.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T18:55:40.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T18:55:40.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T18:55:40.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T18:55:40.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T18:55:40.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T18:55:40.578 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T18:55:40.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T18:55:40.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T18:55:40.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.table_properties_collectors: 2023-07-14T18:55:40.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.inplace_update_support: 0 2023-07-14T18:55:40.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T18:55:40.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T18:55:40.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T18:55:40.579 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T18:55:40.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.bloom_locality: 0 2023-07-14T18:55:40.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.max_successive_merges: 0 2023-07-14T18:55:40.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T18:55:40.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T18:55:40.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T18:55:40.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T18:55:40.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.ttl: 2592000 2023-07-14T18:55:40.580 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T18:55:40.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T18:55:40.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T18:55:40.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.enable_blob_files: false 2023-07-14T18:55:40.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.min_blob_size: 0 2023-07-14T18:55:40.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T18:55:40.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T18:55:40.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T18:55:40.581 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T18:55:40.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T18:55:40.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T18:55:40.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T18:55:40.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T18:55:40.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-07-14T18:55:40.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-07-14T18:55:40.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c1691103-f4f2-47a9-b017-9271e5259f08 2023-07-14T18:55:40.582 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689360940501864, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-07-14T18:55:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-07-14T18:55:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689360940503219, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 92542, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 308, "table_properties": {"data_size": 90564, "index_size": 260, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13263, "raw_average_key_size": 49, "raw_value_size": 83317, "raw_average_value_size": 312, "num_data_blocks": 11, "num_entries": 267, "num_filter_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689360940, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c1691103-f4f2-47a9-b017-9271e5259f08", "db_session_id": "CE3AGZHGL772HQ2MNG4H", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-07-14T18:55:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689360940503311, "job": 1, "event": "recovery_finished"} 2023-07-14T18:55:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-07-14T18:55:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T18:55:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5630d73b0000 2023-07-14T18:55:40.583 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: DB pointer 0x5630d739c000 2023-07-14T18:55:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T18:55:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T18:55:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** DB Stats ** 2023-07-14T18:55:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:55:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T18:55:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:55:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:55:40.584 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T18:55:40.585 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:55:40.585 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:55:40.585 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-07-14T18:55:40.585 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2023-07-14T18:55:40.585 INFO:journalctl@ceph.mon.a.smithi078.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:55:40.585 INFO:journalctl@ceph.mon.a.smithi078.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T18:55:40.585 INFO:journalctl@ceph.mon.a.smithi078.stdout: L0 2/0 92.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:55:40.585 INFO:journalctl@ceph.mon.a.smithi078.stdout: Sum 2/0 92.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:55:40.586 INFO:journalctl@ceph.mon.a.smithi078.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 110.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:55:40.586 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-07-14T18:55:40.586 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** Compaction Stats [default] ** 2023-07-14T18:55:40.586 INFO:journalctl@ceph.mon.a.smithi078.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:55:40.586 INFO:journalctl@ceph.mon.a.smithi078.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T18:55:40.586 INFO:journalctl@ceph.mon.a.smithi078.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 110.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-07-14T18:55:40.586 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-07-14T18:55:40.586 INFO:journalctl@ceph.mon.a.smithi078.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T18:55:40.587 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-07-14T18:55:40.587 INFO:journalctl@ceph.mon.a.smithi078.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:55:40.587 INFO:journalctl@ceph.mon.a.smithi078.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T18:55:40.587 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T18:55:40.587 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T18:55:40.587 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T18:55:40.587 INFO:journalctl@ceph.mon.a.smithi078.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T18:55:40.587 INFO:journalctl@ceph.mon.a.smithi078.stdout: Cumulative compaction: 0.00 GB write, 13.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:55:40.588 INFO:journalctl@ceph.mon.a.smithi078.stdout: Interval compaction: 0.00 GB write, 13.41 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:55:40.588 INFO:journalctl@ceph.mon.a.smithi078.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T18:55:40.588 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache BinnedLRUCache@0x5630d73171f0#7 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-14T18:55:40.588 INFO:journalctl@ceph.mon.a.smithi078.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2023-07-14T18:55:40.588 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-07-14T18:55:40.588 INFO:journalctl@ceph.mon.a.smithi078.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T18:55:40.588 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: starting mon.a rank 0 at public addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] at bind addrs [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:55:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a@-1(???) e1 preinit fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:55:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a@-1(???).mds e1 new map 2023-07-14T18:55:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a@-1(???).mds e1 print_map 2023-07-14T18:55:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout: e1 2023-07-14T18:55:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:55:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:55:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout: legacy client fscid: -1 2023-07-14T18:55:40.589 INFO:journalctl@ceph.mon.a.smithi078.stdout: 2023-07-14T18:55:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout: No filesystems configured 2023-07-14T18:55:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:55:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:55:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:55:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:55:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-07-14T18:55:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta expand map: {default=false} 2023-07-14T18:55:40.590 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta from 'false' to 'false' 2023-07-14T18:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta expanded map: {default=false} 2023-07-14T18:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta expand map: {default=info} 2023-07-14T18:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta from 'info' to 'info' 2023-07-14T18:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta expanded map: {default=info} 2023-07-14T18:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta expand map: {default=daemon} 2023-07-14T18:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T18:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T18:55:40.591 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta expand map: {default=debug} 2023-07-14T18:55:40.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T18:55:40.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: expand_channel_meta expanded map: {default=debug} 2023-07-14T18:55:40.592 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 bash[127385]: 858abc030ff98f491db5b627256bbe9a0244e7d463cec540492890e0b118b3d0 2023-07-14T18:55:40.592 INFO:teuthology.orchestra.run.smithi078.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-07-14T18:55:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 systemd[1]: Started Ceph mon.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T18:55:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mon.a is new leader, mons a in quorum (ranks 0) 2023-07-14T18:55:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: monmap e1: 1 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0]} removed_ranks: {} 2023-07-14T18:55:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: fsmap 2023-07-14T18:55:40.840 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: osdmap e1: 0 total, 0 up, 0 in 2023-07-14T18:55:40.841 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:40 smithi078 ceph-mon[127425]: mgrmap e1: no daemons active 2023-07-14T18:55:41.435 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.017+0000 7faeb2786700 1 Processor -- start 2023-07-14T18:55:41.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.017+0000 7faeb2786700 1 -- start start 2023-07-14T18:55:41.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.017+0000 7faeb2786700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac104980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:41.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.017+0000 7faeb2786700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac104ec0 con 0x7faeac104570 2023-07-14T18:55:41.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.018+0000 7faeabfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac104980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:41.436 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.018+0000 7faeabfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac104980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33132/0 (socket says 172.21.15.78:33132) 2023-07-14T18:55:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.018+0000 7faeabfff700 1 -- 172.21.15.78:0/2264190298 learned_addr learned my addr 172.21.15.78:0/2264190298 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.018+0000 7faeabfff700 1 -- 172.21.15.78:0/2264190298 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeac105000 con 0x7faeac104570 2023-07-14T18:55:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.018+0000 7faeabfff700 1 --2- 172.21.15.78:0/2264190298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac104980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fae940098a0 tx=0x7fae94009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a656310d30068c91 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.018+0000 7faeaaffd700 1 -- 172.21.15.78:0/2264190298 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae94004160 con 0x7faeac104570 2023-07-14T18:55:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeaaffd700 1 -- 172.21.15.78:0/2264190298 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fae940042c0 con 0x7faeac104570 2023-07-14T18:55:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeaaffd700 1 -- 172.21.15.78:0/2264190298 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae94011940 con 0x7faeac104570 2023-07-14T18:55:41.437 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeb2786700 1 -- 172.21.15.78:0/2264190298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 msgr2=0x7faeac104980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:41.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeb2786700 1 --2- 172.21.15.78:0/2264190298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac104980 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fae940098a0 tx=0x7fae94009bb0 comp rx=0 tx=0).stop 2023-07-14T18:55:41.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeb2786700 1 -- 172.21.15.78:0/2264190298 shutdown_connections 2023-07-14T18:55:41.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeb2786700 1 --2- 172.21.15.78:0/2264190298 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac104980 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:41.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeb2786700 1 -- 172.21.15.78:0/2264190298 >> 172.21.15.78:0/2264190298 conn(0x7faeac0fee80 msgr2=0x7faeac101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:41.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeb2786700 1 -- 172.21.15.78:0/2264190298 shutdown_connections 2023-07-14T18:55:41.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.019+0000 7faeb2786700 1 -- 172.21.15.78:0/2264190298 wait complete. 2023-07-14T18:55:41.438 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.020+0000 7faeb2786700 1 Processor -- start 2023-07-14T18:55:41.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.020+0000 7faeb2786700 1 -- start start 2023-07-14T18:55:41.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.020+0000 7faeb2786700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:41.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.020+0000 7faeb2786700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeac1980f0 con 0x7faeac104570 2023-07-14T18:55:41.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.020+0000 7faeabfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:41.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faeabfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac197bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33134/0 (socket says 172.21.15.78:33134) 2023-07-14T18:55:41.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faeabfff700 1 -- 172.21.15.78:0/2356919928 learned_addr learned my addr 172.21.15.78:0/2356919928 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:41.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faeabfff700 1 -- 172.21.15.78:0/2356919928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae94009550 con 0x7faeac104570 2023-07-14T18:55:41.439 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faeabfff700 1 --2- 172.21.15.78:0/2356919928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac197bb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fae94000c00 tx=0x7fae94011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:41.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faea97fa700 1 -- 172.21.15.78:0/2356919928 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae940036a0 con 0x7faeac104570 2023-07-14T18:55:41.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeac1982f0 con 0x7faeac104570 2023-07-14T18:55:41.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faea97fa700 1 -- 172.21.15.78:0/2356919928 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fae9401a560 con 0x7faeac104570 2023-07-14T18:55:41.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faea97fa700 1 -- 172.21.15.78:0/2356919928 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fae940245d0 con 0x7faeac104570 2023-07-14T18:55:41.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeac198710 con 0x7faeac104570 2023-07-14T18:55:41.440 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.021+0000 7faea97fa700 1 -- 172.21.15.78:0/2356919928 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fae94024b50 con 0x7faeac104570 2023-07-14T18:55:41.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.022+0000 7faea97fa700 1 -- 172.21.15.78:0/2356919928 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fae9402c6e0 con 0x7faeac104570 2023-07-14T18:55:41.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.022+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeac061c70 con 0x7faeac104570 2023-07-14T18:55:41.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.024+0000 7faea97fa700 1 -- 172.21.15.78:0/2356919928 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fae9402dd50 con 0x7faeac104570 2023-07-14T18:55:41.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.090+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7faeac105f40 con 0x7faeac104570 2023-07-14T18:55:41.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.093+0000 7faea97fa700 1 -- 172.21.15.78:0/2356919928 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7faeac105f40 con 0x7faeac104570 2023-07-14T18:55:41.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.093+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 msgr2=0x7faeac197bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:41.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.093+0000 7faeb2786700 1 --2- 172.21.15.78:0/2356919928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac197bb0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fae94000c00 tx=0x7fae94011e70 comp rx=0 tx=0).stop 2023-07-14T18:55:41.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.093+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 shutdown_connections 2023-07-14T18:55:41.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.093+0000 7faeb2786700 1 --2- 172.21.15.78:0/2356919928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeac104570 0x7faeac197bb0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:41.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.093+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 >> 172.21.15.78:0/2356919928 conn(0x7faeac0fee80 msgr2=0x7faeac0ffa80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:41.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.093+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 shutdown_connections 2023-07-14T18:55:41.442 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:41.093+0000 7faeb2786700 1 -- 172.21.15.78:0/2356919928 wait complete. 2023-07-14T18:55:41.442 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote config to /etc/ceph/ceph.conf 2023-07-14T18:55:41.481 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:55:41.481 INFO:teuthology.orchestra.run.smithi078.stdout:Creating mgr... 2023-07-14T18:55:41.482 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 9283 ... 2023-07-14T18:55:41.482 INFO:teuthology.orchestra.run.smithi078.stdout:Verifying port 8765 ... 2023-07-14T18:55:41.681 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T18:55:41.681 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.x.service: Unit ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.x.service not loaded. 2023-07-14T18:55:41.770 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c.target.wants/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.x.service → /etc/systemd/system/ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@.service. 2023-07-14T18:55:42.049 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:41 smithi078 systemd[1]: Starting Ceph mgr.x for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T18:55:42.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:42 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2356919928' entity='client.admin' 2023-07-14T18:55:42.359 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:42 smithi078 podman[127654]: 2023-07-14T18:55:42.360 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:42 smithi078 bash[127654]: 0b3018b2a03c4e0190c4263f924bc3c54bab4fb9282ab12445058737db2096c8 2023-07-14T18:55:42.371 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:55:42.371 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2023-07-14T18:55:42.379 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:55:42.379 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-07-14T18:55:42.379 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2023-07-14T18:55:42.379 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to enable service . firewalld.service is not available 2023-07-14T18:55:42.387 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:55:42.387 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2023-07-14T18:55:42.394 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:55:42.394 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-07-14T18:55:42.394 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2023-07-14T18:55:42.395 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-07-14T18:55:42.395 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr to start... 2023-07-14T18:55:42.395 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr... 2023-07-14T18:55:42.616 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:42 smithi078 systemd[1]: Started Ceph mgr.x for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T18:55:42.616 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:42.439+0000 7f5cbb250200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:55:42.616 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:42.519+0000 7f5cbb250200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:55:42.616 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:42.610+0000 7f5cbb250200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:55:42.892 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:42.711+0000 7f5cbb250200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:55:42.893 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:42.886+0000 7f5cbb250200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:55:43.160 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:43 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/4030013794' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:55:43.441 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:43.443 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:55:43.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T18:55:43.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:55:43.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:55:43.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:55:43.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:55:43.444 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:43.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:55:43.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:55:43.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:55:43.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:43.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:55:43.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:55:43.445 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2023-07-14T18:55:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:55:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:55:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:55:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:55:43.446 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:43.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:55:43.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:55:43.447 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:55:43.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:55:43.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:55:43.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:55:43.453 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:43.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:55:43.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:55:43.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:55:43.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:55:43.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:55:43.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:55:43.454 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:55:43.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:55:43.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:55:43.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:43.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:55:43.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:43.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:55:43.455 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:55:43.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:43.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:55:43.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T18:55:43.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:55:43.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:55:43.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:55:43.456 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:55:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:55:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:55:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:55:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:55:36.240772+0000", 2023-07-14T18:55:43.457 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:55:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:55:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:55:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.912+0000 7f56c4197700 1 Processor -- start 2023-07-14T18:55:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.912+0000 7f56c4197700 1 -- start start 2023-07-14T18:55:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.912+0000 7f56c4197700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc1055a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:43.458 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.912+0000 7f56c4197700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc105ae0 con 0x7f56bc105190 2023-07-14T18:55:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.913+0000 7f56c1f33700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc1055a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.913+0000 7f56c1f33700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc1055a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33140/0 (socket says 172.21.15.78:33140) 2023-07-14T18:55:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.913+0000 7f56c1f33700 1 -- 172.21.15.78:0/1268545217 learned_addr learned my addr 172.21.15.78:0/1268545217 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.913+0000 7f56c1f33700 1 -- 172.21.15.78:0/1268545217 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56bc105c20 con 0x7f56bc105190 2023-07-14T18:55:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.913+0000 7f56c1f33700 1 --2- 172.21.15.78:0/1268545217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc1055a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f56b80098a0 tx=0x7f56b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad52d5827d363e33 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.913+0000 7f56c0f31700 1 -- 172.21.15.78:0/1268545217 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56b8004160 con 0x7f56bc105190 2023-07-14T18:55:43.459 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.913+0000 7f56c0f31700 1 -- 172.21.15.78:0/1268545217 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f56b80042c0 con 0x7f56bc105190 2023-07-14T18:55:43.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.913+0000 7f56c0f31700 1 -- 172.21.15.78:0/1268545217 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56b8011940 con 0x7f56bc105190 2023-07-14T18:55:43.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.914+0000 7f56c4197700 1 -- 172.21.15.78:0/1268545217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 msgr2=0x7f56bc1055a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:43.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.914+0000 7f56c4197700 1 --2- 172.21.15.78:0/1268545217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc1055a0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f56b80098a0 tx=0x7f56b8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:55:43.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.914+0000 7f56c4197700 1 -- 172.21.15.78:0/1268545217 shutdown_connections 2023-07-14T18:55:43.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.914+0000 7f56c4197700 1 --2- 172.21.15.78:0/1268545217 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc1055a0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:43.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.914+0000 7f56c4197700 1 -- 172.21.15.78:0/1268545217 >> 172.21.15.78:0/1268545217 conn(0x7f56bc100820 msgr2=0x7f56bc102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:43.460 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.914+0000 7f56c4197700 1 -- 172.21.15.78:0/1268545217 shutdown_connections 2023-07-14T18:55:43.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.914+0000 7f56c4197700 1 -- 172.21.15.78:0/1268545217 wait complete. 2023-07-14T18:55:43.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.915+0000 7f56c4197700 1 Processor -- start 2023-07-14T18:55:43.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.915+0000 7f56c4197700 1 -- start start 2023-07-14T18:55:43.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.915+0000 7f56c4197700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:43.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.915+0000 7f56c4197700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56bc199270 con 0x7f56bc105190 2023-07-14T18:55:43.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.915+0000 7f56c1f33700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:43.461 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.915+0000 7f56c1f33700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33142/0 (socket says 172.21.15.78:33142) 2023-07-14T18:55:43.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.915+0000 7f56c1f33700 1 -- 172.21.15.78:0/4030013794 learned_addr learned my addr 172.21.15.78:0/4030013794 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:43.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.916+0000 7f56c1f33700 1 -- 172.21.15.78:0/4030013794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56b8009550 con 0x7f56bc105190 2023-07-14T18:55:43.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.916+0000 7f56c1f33700 1 --2- 172.21.15.78:0/4030013794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc198d30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f56b8011dc0 tx=0x7f56b8011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:43.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.916+0000 7f56b2ffd700 1 -- 172.21.15.78:0/4030013794 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56b80037d0 con 0x7f56bc105190 2023-07-14T18:55:43.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.916+0000 7f56b2ffd700 1 -- 172.21.15.78:0/4030013794 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f56b801b440 con 0x7f56bc105190 2023-07-14T18:55:43.462 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.916+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56bc199470 con 0x7f56bc105190 2023-07-14T18:55:43.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.916+0000 7f56b2ffd700 1 -- 172.21.15.78:0/4030013794 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56b801a560 con 0x7f56bc105190 2023-07-14T18:55:43.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.916+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56bc199890 con 0x7f56bc105190 2023-07-14T18:55:43.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.917+0000 7f56b2ffd700 1 -- 172.21.15.78:0/4030013794 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f56b801aad0 con 0x7f56bc105190 2023-07-14T18:55:43.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.917+0000 7f56b2ffd700 1 -- 172.21.15.78:0/4030013794 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f56b80193f0 con 0x7f56bc105190 2023-07-14T18:55:43.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.917+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56bc061c70 con 0x7f56bc105190 2023-07-14T18:55:43.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:42.921+0000 7f56b2ffd700 1 -- 172.21.15.78:0/4030013794 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f56b8029060 con 0x7f56bc105190 2023-07-14T18:55:43.463 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.004+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f56bc106ff0 con 0x7f56bc105190 2023-07-14T18:55:43.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.004+0000 7f56b2ffd700 1 -- 172.21.15.78:0/4030013794 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f56b801f070 con 0x7f56bc105190 2023-07-14T18:55:43.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.005+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 msgr2=0x7f56bc198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:43.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.005+0000 7f56c4197700 1 --2- 172.21.15.78:0/4030013794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc198d30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f56b8011dc0 tx=0x7f56b8011ea0 comp rx=0 tx=0).stop 2023-07-14T18:55:43.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.005+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 shutdown_connections 2023-07-14T18:55:43.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.005+0000 7f56c4197700 1 --2- 172.21.15.78:0/4030013794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f56bc105190 0x7f56bc198d30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:43.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.005+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 >> 172.21.15.78:0/4030013794 conn(0x7f56bc100820 msgr2=0x7f56bc101490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:43.464 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.005+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 shutdown_connections 2023-07-14T18:55:43.465 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:43.005+0000 7f56c4197700 1 -- 172.21.15.78:0/4030013794 wait complete. 2023-07-14T18:55:43.465 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (1/15)... 2023-07-14T18:55:43.465 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:43.227+0000 7f5cbb250200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:55:43.465 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:43.316+0000 7f5cbb250200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:55:43.799 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:43.448+0000 7f5cbb250200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:55:43.799 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:43.542+0000 7f5cbb250200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:55:44.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:43.958+0000 7f5cbb250200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:55:44.799 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:44 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:44.525+0000 7f5cbb250200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:55:44.799 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:44 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:44.692+0000 7f5cbb250200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:55:45.737 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:45 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:45.462+0000 7f5cbb250200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:55:45.737 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:45 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:45.556+0000 7f5cbb250200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:55:45.738 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:45 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:45.732+0000 7f5cbb250200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:55:46.023 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:46 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:46.018+0000 7f5cbb250200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:55:46.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:46 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:46.251+0000 7f5cbb250200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:55:46.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:46.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:55:46.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T18:55:46.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:55:46.391 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:55:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:55:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:55:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:55:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:55:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:55:46.392 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:55:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:55:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2023-07-14T18:55:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:55:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:55:46.393 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:55:46.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:46.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:55:46.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:46.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:55:46.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:55:46.394 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:55:46.399 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:55:46.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:55:46.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:55:46.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:46.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:55:46.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:55:46.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:55:46.400 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:55:46.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:55:46.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:55:46.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:55:46.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:55:46.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:55:46.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:46.401 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:55:46.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:46.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:55:46.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:55:46.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:46.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:55:46.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T18:55:46.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:55:46.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:55:46.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:55:46.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:55:46.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:55:46.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:46.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:55:46.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:46.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:55:46.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:46.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:55:36.240772+0000", 2023-07-14T18:55:46.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:55:46.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:46.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:55:46.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:55:46.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.942+0000 7fd4450f7700 1 Processor -- start 2023-07-14T18:55:46.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd4450f7700 1 -- start start 2023-07-14T18:55:46.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd4450f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd4401080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd4450f7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd440079d10 con 0x7fd440105cd0 2023-07-14T18:55:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd43ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd4401080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd43ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd4401080b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33144/0 (socket says 172.21.15.78:33144) 2023-07-14T18:55:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd43ed9d700 1 -- 172.21.15.78:0/1757111722 learned_addr learned my addr 172.21.15.78:0/1757111722 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd43ed9d700 1 -- 172.21.15.78:0/1757111722 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd440079e50 con 0x7fd440105cd0 2023-07-14T18:55:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd43ed9d700 1 --2- 172.21.15.78:0/1757111722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd4401080b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd4300098a0 tx=0x7fd430009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab514b869f28654c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd43dd9b700 1 -- 172.21.15.78:0/1757111722 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd430004160 con 0x7fd440105cd0 2023-07-14T18:55:46.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd43dd9b700 1 -- 172.21.15.78:0/1757111722 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd4300042c0 con 0x7fd440105cd0 2023-07-14T18:55:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.943+0000 7fd43dd9b700 1 -- 172.21.15.78:0/1757111722 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd430011940 con 0x7fd440105cd0 2023-07-14T18:55:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.944+0000 7fd4450f7700 1 -- 172.21.15.78:0/1757111722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 msgr2=0x7fd4401080b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.944+0000 7fd4450f7700 1 --2- 172.21.15.78:0/1757111722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd4401080b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd4300098a0 tx=0x7fd430009bb0 comp rx=0 tx=0).stop 2023-07-14T18:55:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.944+0000 7fd4450f7700 1 -- 172.21.15.78:0/1757111722 shutdown_connections 2023-07-14T18:55:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.944+0000 7fd4450f7700 1 --2- 172.21.15.78:0/1757111722 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd4401080b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.944+0000 7fd4450f7700 1 -- 172.21.15.78:0/1757111722 >> 172.21.15.78:0/1757111722 conn(0x7fd4400ff710 msgr2=0x7fd440101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.944+0000 7fd4450f7700 1 -- 172.21.15.78:0/1757111722 shutdown_connections 2023-07-14T18:55:46.406 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.944+0000 7fd4450f7700 1 -- 172.21.15.78:0/1757111722 wait complete. 2023-07-14T18:55:46.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.945+0000 7fd4450f7700 1 Processor -- start 2023-07-14T18:55:46.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.945+0000 7fd4450f7700 1 -- start start 2023-07-14T18:55:46.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.945+0000 7fd4450f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd440105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:46.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.945+0000 7fd4450f7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd440105b70 con 0x7fd440105cd0 2023-07-14T18:55:46.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.945+0000 7fd43ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd440105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:46.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.945+0000 7fd43ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd440105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33146/0 (socket says 172.21.15.78:33146) 2023-07-14T18:55:46.407 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.945+0000 7fd43ed9d700 1 -- 172.21.15.78:0/1122749794 learned_addr learned my addr 172.21.15.78:0/1122749794 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:46.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd43ed9d700 1 -- 172.21.15.78:0/1122749794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd430009550 con 0x7fd440105cd0 2023-07-14T18:55:46.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd43ed9d700 1 --2- 172.21.15.78:0/1122749794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd440105630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd430000c00 tx=0x7fd430011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:46.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd427fff700 1 -- 172.21.15.78:0/1122749794 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4300036a0 con 0x7fd440105cd0 2023-07-14T18:55:46.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd440103c80 con 0x7fd440105cd0 2023-07-14T18:55:46.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd427fff700 1 -- 172.21.15.78:0/1122749794 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd43001b440 con 0x7fd440105cd0 2023-07-14T18:55:46.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd427fff700 1 -- 172.21.15.78:0/1122749794 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd4300193f0 con 0x7fd440105cd0 2023-07-14T18:55:46.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd440104100 con 0x7fd440105cd0 2023-07-14T18:55:46.408 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd427fff700 1 -- 172.21.15.78:0/1122749794 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd430019920 con 0x7fd440105cd0 2023-07-14T18:55:46.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.946+0000 7fd427fff700 1 -- 172.21.15.78:0/1122749794 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fd43001a3d0 con 0x7fd440105cd0 2023-07-14T18:55:46.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.947+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd440109190 con 0x7fd440105cd0 2023-07-14T18:55:46.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:45.949+0000 7fd427fff700 1 -- 172.21.15.78:0/1122749794 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7fd430029060 con 0x7fd440105cd0 2023-07-14T18:55:46.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.021+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd440104760 con 0x7fd440105cd0 2023-07-14T18:55:46.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.022+0000 7fd427fff700 1 -- 172.21.15.78:0/1122749794 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7fd43001f070 con 0x7fd440105cd0 2023-07-14T18:55:46.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.022+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 msgr2=0x7fd440105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:46.409 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.022+0000 7fd4450f7700 1 --2- 172.21.15.78:0/1122749794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd440105630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd430000c00 tx=0x7fd430011e70 comp rx=0 tx=0).stop 2023-07-14T18:55:46.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.023+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 shutdown_connections 2023-07-14T18:55:46.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.023+0000 7fd4450f7700 1 --2- 172.21.15.78:0/1122749794 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd440105cd0 0x7fd440105630 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:46.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.023+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 >> 172.21.15.78:0/1122749794 conn(0x7fd4400ff710 msgr2=0x7fd4401003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:46.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.023+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 shutdown_connections 2023-07-14T18:55:46.410 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:46.023+0000 7fd4450f7700 1 -- 172.21.15.78:0/1122749794 wait complete. 2023-07-14T18:55:46.410 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (2/15)... 2023-07-14T18:55:46.667 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:46 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1122749794' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:55:46.667 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:46 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:46.401+0000 7f5cbb250200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:55:47.049 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:46 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:46.661+0000 7f5cbb250200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:55:47.672 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:47.351+0000 7f5cbb250200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:55:47.672 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:47.480+0000 7f5cbb250200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:55:47.998 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:47.666+0000 7f5cbb250200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:55:47.999 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:47.778+0000 7f5cbb250200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:55:48.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:47.993+0000 7f5cbb250200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:55:49.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:55:48.863+0000 7f5cbb250200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:55:49.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: Activating manager daemon x 2023-07-14T18:55:49.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: mgrmap e2: x(active, starting, since 0.00509431s) 2023-07-14T18:55:49.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:55:49.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:55:49.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:55:49.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:55:49.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:55:49.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: Manager daemon x is now available 2023-07-14T18:55:49.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:55:49.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:55:49.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' 2023-07-14T18:55:49.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' 2023-07-14T18:55:49.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:48 smithi078 ceph-mon[127425]: from='mgr.14100 172.21.15.78:0/849863476' entity='mgr.x' 2023-07-14T18:55:49.338 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:49.338 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:55:49.339 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T18:55:49.339 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:55:49.339 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:55:49.339 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:55:49.339 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:55:49.339 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:49.339 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:55:49.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:55:49.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:55:49.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:49.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:55:49.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:55:49.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:49.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2023-07-14T18:55:49.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:55:49.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:49.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:55:49.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:55:49.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:49.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:55:49.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:49.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:55:49.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:55:49.347 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:55:49.347 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:55:49.347 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:55:49.347 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:55:49.348 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:49.348 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:55:49.348 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:55:49.348 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:55:49.348 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:55:49.348 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:55:49.348 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:55:49.348 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:55:49.349 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:55:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": false, 2023-07-14T18:55:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:55:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:55:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:55:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:55:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:55:49.350 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:55:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:55:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:55:36.240772+0000", 2023-07-14T18:55:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:55:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:49.351 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:55:49.352 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:55:49.352 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.873+0000 7f855c61c700 1 Processor -- start 2023-07-14T18:55:49.352 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.873+0000 7f855c61c700 1 -- start start 2023-07-14T18:55:49.352 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.873+0000 7f855c61c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f8554103e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:49.352 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.873+0000 7f855c61c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8554079d50 con 0x7f8554103a80 2023-07-14T18:55:49.352 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.873+0000 7f855a3b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f8554103e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:49.352 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.874+0000 7f855a3b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f8554103e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33148/0 (socket says 172.21.15.78:33148) 2023-07-14T18:55:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.874+0000 7f855a3b8700 1 -- 172.21.15.78:0/2430923012 learned_addr learned my addr 172.21.15.78:0/2430923012 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.874+0000 7f855a3b8700 1 -- 172.21.15.78:0/2430923012 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8554079e90 con 0x7f8554103a80 2023-07-14T18:55:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.874+0000 7f855a3b8700 1 --2- 172.21.15.78:0/2430923012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f8554103e90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f854c009e80 tx=0x7f854c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230d2e46e07e1b21 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.874+0000 7f85593b6700 1 -- 172.21.15.78:0/2430923012 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f854c00e070 con 0x7f8554103a80 2023-07-14T18:55:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.874+0000 7f85593b6700 1 -- 172.21.15.78:0/2430923012 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f854c003e90 con 0x7f8554103a80 2023-07-14T18:55:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.874+0000 7f85593b6700 1 -- 172.21.15.78:0/2430923012 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f854c004360 con 0x7f8554103a80 2023-07-14T18:55:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 -- 172.21.15.78:0/2430923012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 msgr2=0x7f8554103e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:49.353 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 --2- 172.21.15.78:0/2430923012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f8554103e90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f854c009e80 tx=0x7f854c00b330 comp rx=0 tx=0).stop 2023-07-14T18:55:49.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 -- 172.21.15.78:0/2430923012 shutdown_connections 2023-07-14T18:55:49.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 --2- 172.21.15.78:0/2430923012 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f8554103e90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:49.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 -- 172.21.15.78:0/2430923012 >> 172.21.15.78:0/2430923012 conn(0x7f85540ff710 msgr2=0x7f8554101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:49.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 -- 172.21.15.78:0/2430923012 shutdown_connections 2023-07-14T18:55:49.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 -- 172.21.15.78:0/2430923012 wait complete. 2023-07-14T18:55:49.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 Processor -- start 2023-07-14T18:55:49.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 -- start start 2023-07-14T18:55:49.354 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f855419af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:49.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.875+0000 7f855c61c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f855419b490 con 0x7f8554103a80 2023-07-14T18:55:49.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f855a3b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f855419af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:49.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f855a3b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f855419af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33150/0 (socket says 172.21.15.78:33150) 2023-07-14T18:55:49.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f855a3b8700 1 -- 172.21.15.78:0/338287531 learned_addr learned my addr 172.21.15.78:0/338287531 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:49.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f855a3b8700 1 -- 172.21.15.78:0/338287531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c009b30 con 0x7f8554103a80 2023-07-14T18:55:49.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f855a3b8700 1 --2- 172.21.15.78:0/338287531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f855419af50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f854c0072a0 tx=0x7f854c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:49.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f854b7fe700 1 -- 172.21.15.78:0/338287531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f854c00e040 con 0x7f8554103a80 2023-07-14T18:55:49.355 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f854b7fe700 1 -- 172.21.15.78:0/338287531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f854c0128a0 con 0x7f8554103a80 2023-07-14T18:55:49.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855419b690 con 0x7f8554103a80 2023-07-14T18:55:49.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f855419bb10 con 0x7f8554103a80 2023-07-14T18:55:49.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f854b7fe700 1 -- 172.21.15.78:0/338287531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f854c01c6b0 con 0x7f8554103a80 2023-07-14T18:55:49.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f854b7fe700 1 -- 172.21.15.78:0/338287531 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f854c01cc70 con 0x7f8554103a80 2023-07-14T18:55:49.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.876+0000 7f854b7fe700 1 -- 172.21.15.78:0/338287531 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f854c01b730 con 0x7f8554103a80 2023-07-14T18:55:49.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.877+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8538005320 con 0x7f8554103a80 2023-07-14T18:55:49.356 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.879+0000 7f854b7fe700 1 -- 172.21.15.78:0/338287531 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72446 (secure 0 0 0) 0x7f854c02a060 con 0x7f8554103a80 2023-07-14T18:55:49.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.880+0000 7f854b7fe700 1 -- 172.21.15.78:0/338287531 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 2) v1 ==== 45239+0+0 (secure 0 0 0) 0x7f854c012430 con 0x7f8554103a80 2023-07-14T18:55:49.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.959+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8538005cc0 con 0x7f8554103a80 2023-07-14T18:55:49.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.960+0000 7f854b7fe700 1 -- 172.21.15.78:0/338287531 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f854c017020 con 0x7f8554103a80 2023-07-14T18:55:49.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.961+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 msgr2=0x7f855419af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:49.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.961+0000 7f855c61c700 1 --2- 172.21.15.78:0/338287531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f855419af50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f854c0072a0 tx=0x7f854c003d00 comp rx=0 tx=0).stop 2023-07-14T18:55:49.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.961+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 shutdown_connections 2023-07-14T18:55:49.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.961+0000 7f855c61c700 1 --2- 172.21.15.78:0/338287531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8554103a80 0x7f855419af50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:49.357 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.961+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 >> 172.21.15.78:0/338287531 conn(0x7f85540ff710 msgr2=0x7f855418f640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:49.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.961+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 shutdown_connections 2023-07-14T18:55:49.358 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:48.961+0000 7f855c61c700 1 -- 172.21.15.78:0/338287531 wait complete. 2023-07-14T18:55:49.358 INFO:teuthology.orchestra.run.smithi078.stdout:mgr not available, waiting (3/15)... 2023-07-14T18:55:50.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:49 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/338287531' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:55:50.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:49 smithi078 ceph-mon[127425]: mgrmap e3: x(active, since 1.00852s) 2023-07-14T18:55:52.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:52 smithi078 ceph-mon[127425]: mgrmap e4: x(active, since 2s) 2023-07-14T18:55:53.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:53 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/214868246' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-07-14T18:55:56.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:55:56.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:55:56.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T18:55:56.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "health": { 2023-07-14T18:55:56.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-07-14T18:55:56.402 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-07-14T18:55:56.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-07-14T18:55:56.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:56.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-07-14T18:55:56.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-07-14T18:55:56.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 0 2023-07-14T18:55:56.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:56.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-07-14T18:55:56.403 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "a" 2023-07-14T18:55:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2023-07-14T18:55:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "monmap": { 2023-07-14T18:55:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-07-14T18:55:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-07-14T18:55:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:56.404 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-07-14T18:55:56.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:56.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-07-14T18:55:56.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-07-14T18:55:56.405 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-07-14T18:55:56.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-07-14T18:55:56.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-07-14T18:55:56.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-07-14T18:55:56.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:56.411 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-07-14T18:55:56.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-07-14T18:55:56.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-07-14T18:55:56.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-07-14T18:55:56.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-07-14T18:55:56.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-07-14T18:55:56.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-07-14T18:55:56.412 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-07-14T18:55:56.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-07-14T18:55:56.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:56.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-07-14T18:55:56.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:56.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-07-14T18:55:56.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-07-14T18:55:56.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:56.413 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-07-14T18:55:56.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:55:56.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-07-14T18:55:56.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modules": [ 2023-07-14T18:55:56.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "iostat", 2023-07-14T18:55:56.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "nfs", 2023-07-14T18:55:56.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "restful" 2023-07-14T18:55:56.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ], 2023-07-14T18:55:56.414 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:55:56.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:56.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-07-14T18:55:56.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-07-14T18:55:56.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "modified": "2023-07-14T18:55:36.240772+0000", 2023-07-14T18:55:56.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "services": {} 2023-07-14T18:55:56.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout }, 2023-07-14T18:55:56.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-07-14T18:55:56.415 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:55:56.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.818+0000 7f81217bb700 1 Processor -- start 2023-07-14T18:55:56.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.818+0000 7f81217bb700 1 -- start start 2023-07-14T18:55:56.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.818+0000 7f81217bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c1080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:56.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.818+0000 7f81217bb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c079cf0 con 0x7f811c105cf0 2023-07-14T18:55:56.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.818+0000 7f811affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c1080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:56.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.818+0000 7f811affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c1080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33176/0 (socket says 172.21.15.78:33176) 2023-07-14T18:55:56.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.818+0000 7f811affd700 1 -- 172.21.15.78:0/3475697594 learned_addr learned my addr 172.21.15.78:0/3475697594 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:56.416 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.818+0000 7f811affd700 1 -- 172.21.15.78:0/3475697594 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c079e30 con 0x7f811c105cf0 2023-07-14T18:55:56.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f811affd700 1 --2- 172.21.15.78:0/3475697594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c1080d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098a0 tx=0x7f810c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99dacfc1f192bd70 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:56.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f8119ffb700 1 -- 172.21.15.78:0/3475697594 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810c004160 con 0x7f811c105cf0 2023-07-14T18:55:56.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f8119ffb700 1 -- 172.21.15.78:0/3475697594 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f810c0042c0 con 0x7f811c105cf0 2023-07-14T18:55:56.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f8119ffb700 1 -- 172.21.15.78:0/3475697594 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810c011940 con 0x7f811c105cf0 2023-07-14T18:55:56.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f81217bb700 1 -- 172.21.15.78:0/3475697594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 msgr2=0x7f811c1080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:56.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f81217bb700 1 --2- 172.21.15.78:0/3475697594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c1080d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f810c0098a0 tx=0x7f810c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:55:56.417 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f81217bb700 1 -- 172.21.15.78:0/3475697594 shutdown_connections 2023-07-14T18:55:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f81217bb700 1 --2- 172.21.15.78:0/3475697594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c1080d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f81217bb700 1 -- 172.21.15.78:0/3475697594 >> 172.21.15.78:0/3475697594 conn(0x7f811c0ff710 msgr2=0x7f811c101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f81217bb700 1 -- 172.21.15.78:0/3475697594 shutdown_connections 2023-07-14T18:55:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.819+0000 7f81217bb700 1 -- 172.21.15.78:0/3475697594 wait complete. 2023-07-14T18:55:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.820+0000 7f81217bb700 1 Processor -- start 2023-07-14T18:55:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.820+0000 7f81217bb700 1 -- start start 2023-07-14T18:55:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.820+0000 7f81217bb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:56.418 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.820+0000 7f81217bb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c199270 con 0x7f811c105cf0 2023-07-14T18:55:56.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.820+0000 7f811affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:56.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.820+0000 7f811affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33178/0 (socket says 172.21.15.78:33178) 2023-07-14T18:55:56.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.820+0000 7f811affd700 1 -- 172.21.15.78:0/214868246 learned_addr learned my addr 172.21.15.78:0/214868246 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:55:56.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.821+0000 7f811affd700 1 -- 172.21.15.78:0/214868246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f810c009550 con 0x7f811c105cf0 2023-07-14T18:55:56.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.821+0000 7f811affd700 1 --2- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c198d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f810c000c00 tx=0x7f810c011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:56.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.821+0000 7f8103fff700 1 -- 172.21.15.78:0/214868246 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810c0036a0 con 0x7f811c105cf0 2023-07-14T18:55:56.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.821+0000 7f8103fff700 1 -- 172.21.15.78:0/214868246 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f810c01b440 con 0x7f811c105cf0 2023-07-14T18:55:56.419 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.821+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811c199470 con 0x7f811c105cf0 2023-07-14T18:55:56.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.821+0000 7f8103fff700 1 -- 172.21.15.78:0/214868246 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f810c0193f0 con 0x7f811c105cf0 2023-07-14T18:55:56.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.821+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811c199890 con 0x7f811c105cf0 2023-07-14T18:55:56.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.822+0000 7f8103fff700 1 -- 172.21.15.78:0/214868246 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f810c019630 con 0x7f811c105cf0 2023-07-14T18:55:56.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.822+0000 7f8103fff700 1 --2- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f8104038780 0x7f810403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:55:56.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.822+0000 7f8103fff700 1 -- 172.21.15.78:0/214868246 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f810c04c5c0 con 0x7f811c105cf0 2023-07-14T18:55:56.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.822+0000 7f811a7fc700 1 --2- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f8104038780 0x7f810403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:55:56.420 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.822+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811c061c70 con 0x7f811c105cf0 2023-07-14T18:55:56.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.822+0000 7f811a7fc700 1 --2- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f8104038780 0x7f810403ac30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8110006fd0 tx=0x7f8110006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:55:56.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:52.827+0000 7f8103fff700 1 -- 172.21.15.78:0/214868246 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f810c01f020 con 0x7f811c105cf0 2023-07-14T18:55:56.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.102+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f811c19c3c0 con 0x7f811c105cf0 2023-07-14T18:55:56.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.102+0000 7f8103fff700 1 -- 172.21.15.78:0/214868246 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1238 (secure 0 0 0) 0x7f810c022950 con 0x7f811c105cf0 2023-07-14T18:55:56.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f8104038780 msgr2=0x7f810403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:56.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 --2- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f8104038780 0x7f810403ac30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8110006fd0 tx=0x7f8110006e40 comp rx=0 tx=0).stop 2023-07-14T18:55:56.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 msgr2=0x7f811c198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:55:56.421 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 --2- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c198d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f810c000c00 tx=0x7f810c011e70 comp rx=0 tx=0).stop 2023-07-14T18:55:56.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 shutdown_connections 2023-07-14T18:55:56.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 --2- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f8104038780 0x7f810403ac30 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:56.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 --2- 172.21.15.78:0/214868246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f811c105cf0 0x7f811c198d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:55:56.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 >> 172.21.15.78:0/214868246 conn(0x7f811c0ff710 msgr2=0x7f811c1003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:55:56.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 shutdown_connections 2023-07-14T18:55:56.422 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:53.104+0000 7f81217bb700 1 -- 172.21.15.78:0/214868246 wait complete. 2023-07-14T18:55:56.422 INFO:teuthology.orchestra.run.smithi078.stdout:mgr is available 2023-07-14T18:55:59.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:55:58 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3561459918' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-07-14T18:56:00.865 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:56:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [global] 2023-07-14T18:56:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout fsid = e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:56:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-07-14T18:56:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-07-14T18:56:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-07-14T18:56:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-07-14T18:56:00.866 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:56:00.867 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [mgr] 2023-07-14T18:56:00.867 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-07-14T18:56:00.867 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 2023-07-14T18:56:00.867 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout [osd] 2023-07-14T18:56:00.867 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2023-07-14T18:56:00.867 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2023-07-14T18:56:00.867 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2023-07-14T18:56:00.867 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-07-14T18:56:00.868 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-07-14T18:56:00.868 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.421+0000 7f5fd6738700 1 Processor -- start 2023-07-14T18:56:00.868 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.421+0000 7f5fd6738700 1 -- start start 2023-07-14T18:56:00.868 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.421+0000 7f5fd6738700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd01049a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:00.868 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.421+0000 7f5fd6738700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd0104ee0 con 0x7f5fd0104590 2023-07-14T18:56:00.868 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.422+0000 7f5fcffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd01049a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:00.868 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.422+0000 7f5fcffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd01049a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33182/0 (socket says 172.21.15.78:33182) 2023-07-14T18:56:00.869 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.422+0000 7f5fcffff700 1 -- 172.21.15.78:0/4253270760 learned_addr learned my addr 172.21.15.78:0/4253270760 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:00.869 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.422+0000 7f5fcffff700 1 -- 172.21.15.78:0/4253270760 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd0105020 con 0x7f5fd0104590 2023-07-14T18:56:00.869 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.422+0000 7f5fcffff700 1 --2- 172.21.15.78:0/4253270760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd01049a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00098a0 tx=0x7f5fc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c9f8cff6b46e5f6 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:00.869 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.422+0000 7f5fceffd700 1 -- 172.21.15.78:0/4253270760 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5fc0004160 con 0x7f5fd0104590 2023-07-14T18:56:00.869 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.422+0000 7f5fceffd700 1 -- 172.21.15.78:0/4253270760 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5fc00042c0 con 0x7f5fd0104590 2023-07-14T18:56:00.869 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.422+0000 7f5fceffd700 1 -- 172.21.15.78:0/4253270760 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5fc0011940 con 0x7f5fd0104590 2023-07-14T18:56:00.869 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 -- 172.21.15.78:0/4253270760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 msgr2=0x7f5fd01049a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:00.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 --2- 172.21.15.78:0/4253270760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd01049a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5fc00098a0 tx=0x7f5fc0009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:00.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 -- 172.21.15.78:0/4253270760 shutdown_connections 2023-07-14T18:56:00.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 --2- 172.21.15.78:0/4253270760 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd01049a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:00.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 -- 172.21.15.78:0/4253270760 >> 172.21.15.78:0/4253270760 conn(0x7f5fd00fee80 msgr2=0x7f5fd0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:00.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 -- 172.21.15.78:0/4253270760 shutdown_connections 2023-07-14T18:56:00.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 -- 172.21.15.78:0/4253270760 wait complete. 2023-07-14T18:56:00.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 Processor -- start 2023-07-14T18:56:00.870 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.423+0000 7f5fd6738700 1 -- start start 2023-07-14T18:56:00.871 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fd6738700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd0198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:00.871 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fd6738700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd01992d0 con 0x7f5fd0104590 2023-07-14T18:56:00.871 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fcffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd0198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:00.871 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fcffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd0198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33184/0 (socket says 172.21.15.78:33184) 2023-07-14T18:56:00.871 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fcffff700 1 -- 172.21.15.78:0/3561459918 learned_addr learned my addr 172.21.15.78:0/3561459918 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:00.871 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fcffff700 1 -- 172.21.15.78:0/3561459918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc0009550 con 0x7f5fd0104590 2023-07-14T18:56:00.871 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fcffff700 1 --2- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd0198d90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0000c00 tx=0x7f5fc0011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:00.872 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fcd7fa700 1 -- 172.21.15.78:0/3561459918 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5fc00036a0 con 0x7f5fd0104590 2023-07-14T18:56:00.872 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fcd7fa700 1 -- 172.21.15.78:0/3561459918 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f5fc001b440 con 0x7f5fd0104590 2023-07-14T18:56:00.872 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd01994d0 con 0x7f5fd0104590 2023-07-14T18:56:00.872 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fcd7fa700 1 -- 172.21.15.78:0/3561459918 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5fc00193f0 con 0x7f5fd0104590 2023-07-14T18:56:00.872 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.424+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd01998f0 con 0x7f5fd0104590 2023-07-14T18:56:00.872 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.425+0000 7f5fcd7fa700 1 -- 172.21.15.78:0/3561459918 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f5fc0019630 con 0x7f5fd0104590 2023-07-14T18:56:00.872 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.425+0000 7f5fcd7fa700 1 --2- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f5fbc038780 0x7f5fbc03ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:00.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.425+0000 7f5fcd7fa700 1 -- 172.21.15.78:0/3561459918 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f5fc004c5d0 con 0x7f5fd0104590 2023-07-14T18:56:00.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.425+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd0061c70 con 0x7f5fd0104590 2023-07-14T18:56:00.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.425+0000 7f5fcf7fe700 1 --2- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f5fbc038780 0x7f5fbc03ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:00.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.426+0000 7f5fcf7fe700 1 --2- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f5fbc038780 0x7f5fbc03ac30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8006fd0 tx=0x7f5fb8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:00.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.430+0000 7f5fcd7fa700 1 -- 172.21.15.78:0/3561459918 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5fc001f020 con 0x7f5fd0104590 2023-07-14T18:56:00.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.617+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f5fd019c710 con 0x7f5fd0104590 2023-07-14T18:56:00.873 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.618+0000 7f5fcd7fa700 1 -- 172.21.15.78:0/3561459918 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f5fc0022950 con 0x7f5fd0104590 2023-07-14T18:56:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f5fbc038780 msgr2=0x7f5fbc03ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 --2- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f5fbc038780 0x7f5fbc03ac30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8006fd0 tx=0x7f5fb8006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 msgr2=0x7f5fd0198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 --2- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd0198d90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0000c00 tx=0x7f5fc0011e70 comp rx=0 tx=0).stop 2023-07-14T18:56:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 shutdown_connections 2023-07-14T18:56:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 --2- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f5fbc038780 0x7f5fbc03ac30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 --2- 172.21.15.78:0/3561459918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5fd0104590 0x7f5fd0198d90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:00.874 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 >> 172.21.15.78:0/3561459918 conn(0x7f5fd00fee80 msgr2=0x7f5fd0103ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:00.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 shutdown_connections 2023-07-14T18:56:00.875 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:55:58.620+0000 7f5fd6738700 1 -- 172.21.15.78:0/3561459918 wait complete. 2023-07-14T18:56:00.875 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling cephadm module... 2023-07-14T18:56:01.944 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:01 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/320192184' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-07-14T18:56:01.945 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: ignoring --setuser ceph since I am not root 2023-07-14T18:56:01.945 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: ignoring --setgroup ceph since I am not root 2023-07-14T18:56:01.945 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:01.793+0000 7f7e609ae200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:56:01.945 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:01.873+0000 7f7e609ae200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:56:02.010 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.333+0000 7f4866222700 1 Processor -- start 2023-07-14T18:56:02.014 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.333+0000 7f4866222700 1 -- start start 2023-07-14T18:56:02.015 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.333+0000 7f4866222700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f4860107c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:02.015 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.333+0000 7f4866222700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4860108140 con 0x7f48601077f0 2023-07-14T18:56:02.015 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.333+0000 7f485f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f4860107c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:02.015 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.333+0000 7f485f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f4860107c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33188/0 (socket says 172.21.15.78:33188) 2023-07-14T18:56:02.016 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.333+0000 7f485f7fe700 1 -- 172.21.15.78:0/2416655808 learned_addr learned my addr 172.21.15.78:0/2416655808 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:02.016 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.333+0000 7f485f7fe700 1 -- 172.21.15.78:0/2416655808 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860108280 con 0x7f48601077f0 2023-07-14T18:56:02.016 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f485f7fe700 1 --2- 172.21.15.78:0/2416655808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f4860107c00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f48500098a0 tx=0x7f4850009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e8d8907c22f095 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:02.017 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f485e7fc700 1 -- 172.21.15.78:0/2416655808 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4850004160 con 0x7f48601077f0 2023-07-14T18:56:02.017 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f485e7fc700 1 -- 172.21.15.78:0/2416655808 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f48500042c0 con 0x7f48601077f0 2023-07-14T18:56:02.017 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f485e7fc700 1 -- 172.21.15.78:0/2416655808 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4850011940 con 0x7f48601077f0 2023-07-14T18:56:02.017 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f4866222700 1 -- 172.21.15.78:0/2416655808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 msgr2=0x7f4860107c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:02.018 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f4866222700 1 --2- 172.21.15.78:0/2416655808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f4860107c00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f48500098a0 tx=0x7f4850009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:02.018 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f4866222700 1 -- 172.21.15.78:0/2416655808 shutdown_connections 2023-07-14T18:56:02.018 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f4866222700 1 --2- 172.21.15.78:0/2416655808 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f4860107c00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:02.018 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f4866222700 1 -- 172.21.15.78:0/2416655808 >> 172.21.15.78:0/2416655808 conn(0x7f486007a800 msgr2=0x7f486007ac00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:02.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f4866222700 1 -- 172.21.15.78:0/2416655808 shutdown_connections 2023-07-14T18:56:02.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.334+0000 7f4866222700 1 -- 172.21.15.78:0/2416655808 wait complete. 2023-07-14T18:56:02.019 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.335+0000 7f4866222700 1 Processor -- start 2023-07-14T18:56:02.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.335+0000 7f4866222700 1 -- start start 2023-07-14T18:56:02.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.335+0000 7f4866222700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f486019ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:02.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.335+0000 7f4866222700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f486019b3d0 con 0x7f48601077f0 2023-07-14T18:56:02.020 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.335+0000 7f485f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f486019ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.335+0000 7f485f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f486019ae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33190/0 (socket says 172.21.15.78:33190) 2023-07-14T18:56:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.335+0000 7f485f7fe700 1 -- 172.21.15.78:0/320192184 learned_addr learned my addr 172.21.15.78:0/320192184 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.336+0000 7f485f7fe700 1 -- 172.21.15.78:0/320192184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4850009550 con 0x7f48601077f0 2023-07-14T18:56:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.336+0000 7f485f7fe700 1 --2- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f486019ae90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4850006af0 tx=0x7f4850011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.336+0000 7f485cff9700 1 -- 172.21.15.78:0/320192184 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4850003bf0 con 0x7f48601077f0 2023-07-14T18:56:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.336+0000 7f485cff9700 1 -- 172.21.15.78:0/320192184 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f485001b8b0 con 0x7f48601077f0 2023-07-14T18:56:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.336+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486019b5d0 con 0x7f48601077f0 2023-07-14T18:56:02.021 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.336+0000 7f485cff9700 1 -- 172.21.15.78:0/320192184 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4850019880 con 0x7f48601077f0 2023-07-14T18:56:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.336+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486019b9f0 con 0x7f48601077f0 2023-07-14T18:56:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.337+0000 7f485cff9700 1 -- 172.21.15.78:0/320192184 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 4) v1 ==== 45472+0+0 (secure 0 0 0) 0x7f4850019af0 con 0x7f48601077f0 2023-07-14T18:56:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.337+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4860061c70 con 0x7f48601077f0 2023-07-14T18:56:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.337+0000 7f485cff9700 1 --2- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f4848038780 0x7f484803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.337+0000 7f485cff9700 1 -- 172.21.15.78:0/320192184 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f485004c980 con 0x7f48601077f0 2023-07-14T18:56:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.337+0000 7f485effd700 1 --2- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f4848038780 0x7f484803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:02.022 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.337+0000 7f485effd700 1 --2- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f4848038780 0x7f484803ac30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4854006fd0 tx=0x7f4854006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:02.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.342+0000 7f485cff9700 1 -- 172.21.15.78:0/320192184 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f48500189a0 con 0x7f48601077f0 2023-07-14T18:56:02.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.578+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f486019eb90 con 0x7f48601077f0 2023-07-14T18:56:02.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.631+0000 7f485cff9700 1 -- 172.21.15.78:0/320192184 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f485000fd50 con 0x7f48601077f0 2023-07-14T18:56:02.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.632+0000 7f485cff9700 1 -- 172.21.15.78:0/320192184 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f485004c3a0 con 0x7f48601077f0 2023-07-14T18:56:02.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f4848038780 msgr2=0x7f484803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:02.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 --2- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f4848038780 0x7f484803ac30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4854006fd0 tx=0x7f4854006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:02.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 msgr2=0x7f486019ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:02.023 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 --2- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f486019ae90 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f4850006af0 tx=0x7f4850011a30 comp rx=0 tx=0).stop 2023-07-14T18:56:02.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 shutdown_connections 2023-07-14T18:56:02.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 --2- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f4848038780 0x7f484803ac30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:02.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 --2- 172.21.15.78:0/320192184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48601077f0 0x7f486019ae90 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:02.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 >> 172.21.15.78:0/320192184 conn(0x7f486007a800 msgr2=0x7f4860105590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:02.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.634+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 shutdown_connections 2023-07-14T18:56:02.024 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:01.635+0000 7f4866222700 1 -- 172.21.15.78:0/320192184 wait complete. 2023-07-14T18:56:02.240 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:01.965+0000 7f7e609ae200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:56:02.241 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:02.066+0000 7f7e609ae200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:56:02.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:02.241+0000 7f7e609ae200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:56:02.903 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:02 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/320192184' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-07-14T18:56:02.904 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:02 smithi078 ceph-mon[127425]: mgrmap e5: x(active, since 12s) 2023-07-14T18:56:02.904 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:02.569+0000 7f7e609ae200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:56:02.904 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:02.659+0000 7f7e609ae200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:56:02.904 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:02.791+0000 7f7e609ae200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-07-14T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:56:03.157 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T18:56:03.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T18:56:03.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:56:03.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.505+0000 7f2dfdff4700 1 Processor -- start 2023-07-14T18:56:03.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.505+0000 7f2dfdff4700 1 -- start start 2023-07-14T18:56:03.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.505+0000 7f2dfdff4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df8103890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:03.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2dfdff4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df8103dd0 con 0x7f2df8103480 2023-07-14T18:56:03.158 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2df77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df8103890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:03.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2df77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df8103890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33204/0 (socket says 172.21.15.78:33204) 2023-07-14T18:56:03.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2df77fe700 1 -- 172.21.15.78:0/1481369370 learned_addr learned my addr 172.21.15.78:0/1481369370 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:03.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2df77fe700 1 -- 172.21.15.78:0/1481369370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df8103f10 con 0x7f2df8103480 2023-07-14T18:56:03.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2df77fe700 1 --2- 172.21.15.78:0/1481369370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df8103890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2de80098a0 tx=0x7f2de8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3eafd69f19a86c35 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:03.159 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2df67fc700 1 -- 172.21.15.78:0/1481369370 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2de8004160 con 0x7f2df8103480 2023-07-14T18:56:03.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2df67fc700 1 -- 172.21.15.78:0/1481369370 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2de80042c0 con 0x7f2df8103480 2023-07-14T18:56:03.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.506+0000 7f2df67fc700 1 -- 172.21.15.78:0/1481369370 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2de8011940 con 0x7f2df8103480 2023-07-14T18:56:03.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.507+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1481369370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 msgr2=0x7f2df8103890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:03.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.507+0000 7f2dfdff4700 1 --2- 172.21.15.78:0/1481369370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df8103890 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2de80098a0 tx=0x7f2de8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:03.163 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.507+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1481369370 shutdown_connections 2023-07-14T18:56:03.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.507+0000 7f2dfdff4700 1 --2- 172.21.15.78:0/1481369370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df8103890 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:03.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.507+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1481369370 >> 172.21.15.78:0/1481369370 conn(0x7f2df80fee80 msgr2=0x7f2df8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:03.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.507+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1481369370 shutdown_connections 2023-07-14T18:56:03.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.507+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1481369370 wait complete. 2023-07-14T18:56:03.164 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.508+0000 7f2dfdff4700 1 Processor -- start 2023-07-14T18:56:03.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.508+0000 7f2dfdff4700 1 -- start start 2023-07-14T18:56:03.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.508+0000 7f2dfdff4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df819af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:03.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.508+0000 7f2dfdff4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df819b4c0 con 0x7f2df8103480 2023-07-14T18:56:03.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.508+0000 7f2df77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df819af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:03.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.508+0000 7f2df77fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df819af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33206/0 (socket says 172.21.15.78:33206) 2023-07-14T18:56:03.165 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.508+0000 7f2df77fe700 1 -- 172.21.15.78:0/1807939483 learned_addr learned my addr 172.21.15.78:0/1807939483 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:03.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.508+0000 7f2df77fe700 1 -- 172.21.15.78:0/1807939483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2de8009550 con 0x7f2df8103480 2023-07-14T18:56:03.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.509+0000 7f2df77fe700 1 --2- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df819af80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2de8000c00 tx=0x7f2de8011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:03.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.509+0000 7f2df4ff9700 1 -- 172.21.15.78:0/1807939483 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2de80036a0 con 0x7f2df8103480 2023-07-14T18:56:03.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.509+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df819b6c0 con 0x7f2df8103480 2023-07-14T18:56:03.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.509+0000 7f2df4ff9700 1 -- 172.21.15.78:0/1807939483 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f2de801b440 con 0x7f2df8103480 2023-07-14T18:56:03.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.509+0000 7f2df4ff9700 1 -- 172.21.15.78:0/1807939483 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2de80193f0 con 0x7f2df8103480 2023-07-14T18:56:03.166 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.509+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df819bae0 con 0x7f2df8103480 2023-07-14T18:56:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.510+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df8105460 con 0x7f2df8103480 2023-07-14T18:56:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.510+0000 7f2df4ff9700 1 -- 172.21.15.78:0/1807939483 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2de801b5b0 con 0x7f2df8103480 2023-07-14T18:56:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.510+0000 7f2df4ff9700 1 --2- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f2de00387d0 0x7f2de003ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.510+0000 7f2df6ffd700 1 -- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f2de00387d0 msgr2=0x7f2de003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3249522727 2023-07-14T18:56:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.510+0000 7f2df4ff9700 1 -- 172.21.15.78:0/1807939483 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f2de804c5f0 con 0x7f2df8103480 2023-07-14T18:56:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.510+0000 7f2df6ffd700 1 --2- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f2de00387d0 0x7f2de003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:56:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.519+0000 7f2df4ff9700 1 -- 172.21.15.78:0/1807939483 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2df8105460 con 0x7f2df8103480 2023-07-14T18:56:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.710+0000 7f2df6ffd700 1 -- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f2de00387d0 msgr2=0x7f2de003ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3249522727 2023-07-14T18:56:03.168 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.710+0000 7f2df6ffd700 1 --2- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f2de00387d0 0x7f2de003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:56:03.168 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.760+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2df8105460 con 0x7f2df8103480 2023-07-14T18:56:03.168 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.760+0000 7f2df4ff9700 1 -- 172.21.15.78:0/1807939483 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f2df8105460 con 0x7f2df8103480 2023-07-14T18:56:03.168 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f2de00387d0 msgr2=0x7f2de003ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:56:03.168 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 --2- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f2de00387d0 0x7f2de003ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:03.168 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 msgr2=0x7f2df819af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:03.168 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 --2- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df819af80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2de8000c00 tx=0x7f2de8011e70 comp rx=0 tx=0).stop 2023-07-14T18:56:03.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 shutdown_connections 2023-07-14T18:56:03.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 --2- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7f2de00387d0 0x7f2de003ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:03.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 --2- 172.21.15.78:0/1807939483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df8103480 0x7f2df819af80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:03.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 >> 172.21.15.78:0/1807939483 conn(0x7f2df80fee80 msgr2=0x7f2df8191e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:03.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 shutdown_connections 2023-07-14T18:56:03.169 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:02.763+0000 7f2dfdff4700 1 -- 172.21.15.78:0/1807939483 wait complete. 2023-07-14T18:56:03.169 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2023-07-14T18:56:03.170 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 5... 2023-07-14T18:56:03.170 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:02.896+0000 7f7e609ae200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:56:03.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:03 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:03.328+0000 7f7e609ae200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:56:03.912 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:03 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1807939483' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T18:56:04.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:03 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:03.905+0000 7f7e609ae200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:56:04.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:04.074+0000 7f7e609ae200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:56:05.097 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:04.831+0000 7f7e609ae200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:56:05.098 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:04.920+0000 7f7e609ae200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:56:05.098 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:05 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:05.091+0000 7f7e609ae200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:56:05.742 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:05 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:05.365+0000 7f7e609ae200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:56:05.742 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:05 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:05.590+0000 7f7e609ae200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:56:05.742 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:05 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:05.735+0000 7f7e609ae200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:56:06.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:05 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:05.993+0000 7f7e609ae200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:56:07.010 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:06 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:06.685+0000 7f7e609ae200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:56:07.010 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:06 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:06.814+0000 7f7e609ae200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:56:07.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:07 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:07.003+0000 7f7e609ae200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:56:07.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:07 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:07.116+0000 7f7e609ae200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:56:07.799 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:07 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:07.331+0000 7f7e609ae200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:56:08.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:08.176+0000 7f7e609ae200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:56:08.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: Active manager daemon x restarted 2023-07-14T18:56:08.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: Activating manager daemon x 2023-07-14T18:56:08.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: osdmap e2: 0 total, 0 up, 0 in 2023-07-14T18:56:08.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: mgrmap e6: x(active, starting, since 0.00326382s) 2023-07-14T18:56:08.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:56:08.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:56:08.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:56:08.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:56:08.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:56:08.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: Manager daemon x is now available 2023-07-14T18:56:08.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:08.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:08.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:08.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:08.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:08 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:09.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:09 smithi078 ceph-mon[127425]: Found migration_current of "None". Setting to last migration. 2023-07-14T18:56:09.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:09 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:56:09.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:09 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:56:09.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:09 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:09.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:09 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:09.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:09 smithi078 ceph-mon[127425]: mgrmap e7: x(active, since 1.00499s) 2023-07-14T18:56:09.616 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:56:09.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-07-14T18:56:09.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T18:56:09.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:56:09.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.612+0000 7fc7ae98c700 1 Processor -- start 2023-07-14T18:56:09.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.612+0000 7fc7ae98c700 1 -- start start 2023-07-14T18:56:09.617 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.612+0000 7fc7ae98c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8107d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:09.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.612+0000 7fc7ae98c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a81082a0 con 0x7fc7a8079c90 2023-07-14T18:56:09.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.613+0000 7fc7a7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8107d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:09.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.613+0000 7fc7a7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8107d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33214/0 (socket says 172.21.15.78:33214) 2023-07-14T18:56:09.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.613+0000 7fc7a7fff700 1 -- 172.21.15.78:0/1260414700 learned_addr learned my addr 172.21.15.78:0/1260414700 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:09.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.613+0000 7fc7a7fff700 1 -- 172.21.15.78:0/1260414700 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a81083e0 con 0x7fc7a8079c90 2023-07-14T18:56:09.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.613+0000 7fc7a7fff700 1 --2- 172.21.15.78:0/1260414700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8107d60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc7980098a0 tx=0x7fc798009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=567675089c228d3e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:09.618 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.613+0000 7fc7a6ffd700 1 -- 172.21.15.78:0/1260414700 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc798004160 con 0x7fc7a8079c90 2023-07-14T18:56:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.613+0000 7fc7a6ffd700 1 -- 172.21.15.78:0/1260414700 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc7980042c0 con 0x7fc7a8079c90 2023-07-14T18:56:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.613+0000 7fc7a6ffd700 1 -- 172.21.15.78:0/1260414700 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc798011940 con 0x7fc7a8079c90 2023-07-14T18:56:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.614+0000 7fc7ae98c700 1 -- 172.21.15.78:0/1260414700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 msgr2=0x7fc7a8107d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.614+0000 7fc7ae98c700 1 --2- 172.21.15.78:0/1260414700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8107d60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fc7980098a0 tx=0x7fc798009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.614+0000 7fc7ae98c700 1 -- 172.21.15.78:0/1260414700 shutdown_connections 2023-07-14T18:56:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.614+0000 7fc7ae98c700 1 --2- 172.21.15.78:0/1260414700 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8107d60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.614+0000 7fc7ae98c700 1 -- 172.21.15.78:0/1260414700 >> 172.21.15.78:0/1260414700 conn(0x7fc7a80ff710 msgr2=0x7fc7a8101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:09.619 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.614+0000 7fc7ae98c700 1 -- 172.21.15.78:0/1260414700 shutdown_connections 2023-07-14T18:56:09.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.614+0000 7fc7ae98c700 1 -- 172.21.15.78:0/1260414700 wait complete. 2023-07-14T18:56:09.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.615+0000 7fc7ae98c700 1 Processor -- start 2023-07-14T18:56:09.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.615+0000 7fc7ae98c700 1 -- start start 2023-07-14T18:56:09.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.615+0000 7fc7ae98c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8198d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:09.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.615+0000 7fc7ae98c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8199290 con 0x7fc7a8079c90 2023-07-14T18:56:09.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.615+0000 7fc7a7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8198d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:09.620 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.615+0000 7fc7a7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8198d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33216/0 (socket says 172.21.15.78:33216) 2023-07-14T18:56:09.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.615+0000 7fc7a7fff700 1 -- 172.21.15.78:0/4159252615 learned_addr learned my addr 172.21.15.78:0/4159252615 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:09.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.616+0000 7fc7a7fff700 1 -- 172.21.15.78:0/4159252615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798009550 con 0x7fc7a8079c90 2023-07-14T18:56:09.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.616+0000 7fc7a7fff700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8198d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc798006af0 tx=0x7fc798011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:09.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.616+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc798003c30 con 0x7fc7a8079c90 2023-07-14T18:56:09.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.616+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc79801b8b0 con 0x7fc7a8079c90 2023-07-14T18:56:09.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.616+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a8199490 con 0x7fc7a8079c90 2023-07-14T18:56:09.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.616+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc7980237d0 con 0x7fc7a8079c90 2023-07-14T18:56:09.621 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.616+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a81998b0 con 0x7fc7a8079c90 2023-07-14T18:56:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.617+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 5) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc798023a90 con 0x7fc7a8079c90 2023-07-14T18:56:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.617+0000 7fc7a57fa700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 0x7fc79003ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.617+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 --> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc79003b340 con 0x7fc790038780 2023-07-14T18:56:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.617+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fc79804cc10 con 0x7fc7a8079c90 2023-07-14T18:56:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.617+0000 7fc7a77fe700 1 -- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 msgr2=0x7fc79003ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3249522727 2023-07-14T18:56:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.617+0000 7fc7a77fe700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 0x7fc79003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:56:09.622 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.818+0000 7fc7a77fe700 1 -- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 msgr2=0x7fc79003ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3249522727 2023-07-14T18:56:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:03.818+0000 7fc7a77fe700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 0x7fc79003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:56:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:04.219+0000 7fc7a77fe700 1 -- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 msgr2=0x7fc79003ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3249522727 2023-07-14T18:56:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:04.219+0000 7fc7a77fe700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 0x7fc79003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T18:56:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:05.020+0000 7fc7a77fe700 1 -- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 msgr2=0x7fc79003ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3249522727 2023-07-14T18:56:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:05.020+0000 7fc7a77fe700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 0x7fc79003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T18:56:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:06.621+0000 7fc7a77fe700 1 -- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 msgr2=0x7fc79003ac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/3249522727 2023-07-14T18:56:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:06.621+0000 7fc7a77fe700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 0x7fc79003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T18:56:09.623 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:08.191+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 6) v1 ==== 45250+0+0 (secure 0 0 0) 0x7fc79802c3f0 con 0x7fc7a8079c90 2023-07-14T18:56:09.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:08.191+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 msgr2=0x7fc79003ac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:56:09.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:08.191+0000 7fc7a57fa700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 0x7fc79003ac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:09.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.192+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fc79804e5a0 con 0x7fc7a8079c90 2023-07-14T18:56:09.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.192+0000 7fc7a57fa700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc79003bfb0 0x7fc79003e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:09.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.192+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc79003b340 con 0x7fc79003bfb0 2023-07-14T18:56:09.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.192+0000 7fc7a77fe700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc79003bfb0 0x7fc79003e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:09.624 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.193+0000 7fc7a77fe700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc79003bfb0 0x7fc79003e390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc79c003a10 tx=0x7fc79c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:09.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.194+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mgr.14120 v2:172.21.15.78:6800/378103227 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fc79003b340 con 0x7fc79003bfb0 2023-07-14T18:56:09.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.199+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc7a8061c70 con 0x7fc79003bfb0 2023-07-14T18:56:09.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7a57fa700 1 -- 172.21.15.78:0/4159252615 <== mgr.14120 v2:172.21.15.78:6800/378103227 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc7a8061c70 con 0x7fc79003bfb0 2023-07-14T18:56:09.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc79003bfb0 msgr2=0x7fc79003e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:09.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc79003bfb0 0x7fc79003e390 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fc79c003a10 tx=0x7fc79c009290 comp rx=0 tx=0).stop 2023-07-14T18:56:09.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 msgr2=0x7fc7a8198d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:09.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8198d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fc798006af0 tx=0x7fc798011a30 comp rx=0 tx=0).stop 2023-07-14T18:56:09.625 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 shutdown_connections 2023-07-14T18:56:09.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc79003bfb0 0x7fc79003e390 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:09.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc7a8079c90 0x7fc7a8198d50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:09.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 --2- 172.21.15.78:0/4159252615 >> [v2:172.21.15.78:6800/3249522727,v1:172.21.15.78:6801/3249522727] conn(0x7fc790038780 0x7fc79003ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:09.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 >> 172.21.15.78:0/4159252615 conn(0x7fc7a80ff710 msgr2=0x7fc7a81003e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:09.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 shutdown_connections 2023-07-14T18:56:09.626 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:09.200+0000 7fc7ae98c700 1 -- 172.21.15.78:0/4159252615 wait complete. 2023-07-14T18:56:09.626 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 5 is available 2023-07-14T18:56:09.627 INFO:teuthology.orchestra.run.smithi078.stdout:Setting orchestrator backend to cephadm... 2023-07-14T18:56:10.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:10 smithi078 ceph-mon[127425]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T18:56:10.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:10 smithi078 ceph-mon[127425]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T18:56:10.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:10 smithi078 ceph-mon[127425]: [14/Jul/2023:18:56:09] ENGINE Bus STARTING 2023-07-14T18:56:10.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:10 smithi078 ceph-mon[127425]: [14/Jul/2023:18:56:09] ENGINE Serving on http://172.21.15.78:8765 2023-07-14T18:56:10.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:10 smithi078 ceph-mon[127425]: [14/Jul/2023:18:56:09] ENGINE Serving on https://172.21.15.78:7150 2023-07-14T18:56:10.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:10 smithi078 ceph-mon[127425]: [14/Jul/2023:18:56:09] ENGINE Bus STARTED 2023-07-14T18:56:10.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:10 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:10.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.071+0000 7fd0bb445700 1 Processor -- start 2023-07-14T18:56:10.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0bb445700 1 -- start start 2023-07-14T18:56:10.657 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0bb445700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b41080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:10.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0bb445700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b4079d10 con 0x7fd0b4105cc0 2023-07-14T18:56:10.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0b91e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b41080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:10.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0b91e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b41080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33270/0 (socket says 172.21.15.78:33270) 2023-07-14T18:56:10.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0b91e1700 1 -- 172.21.15.78:0/392263252 learned_addr learned my addr 172.21.15.78:0/392263252 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:10.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0b91e1700 1 -- 172.21.15.78:0/392263252 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b4079e50 con 0x7fd0b4105cc0 2023-07-14T18:56:10.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0b91e1700 1 --2- 172.21.15.78:0/392263252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b41080a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40098a0 tx=0x7fd0a4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b40b7d83ec9ae29 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:10.658 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.072+0000 7fd0abfff700 1 -- 172.21.15.78:0/392263252 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0a4004160 con 0x7fd0b4105cc0 2023-07-14T18:56:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0abfff700 1 -- 172.21.15.78:0/392263252 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd0a40042c0 con 0x7fd0b4105cc0 2023-07-14T18:56:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0abfff700 1 -- 172.21.15.78:0/392263252 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0a4011940 con 0x7fd0b4105cc0 2023-07-14T18:56:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0bb445700 1 -- 172.21.15.78:0/392263252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 msgr2=0x7fd0b41080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0bb445700 1 --2- 172.21.15.78:0/392263252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b41080a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd0a40098a0 tx=0x7fd0a4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0bb445700 1 -- 172.21.15.78:0/392263252 shutdown_connections 2023-07-14T18:56:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0bb445700 1 --2- 172.21.15.78:0/392263252 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b41080a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0bb445700 1 -- 172.21.15.78:0/392263252 >> 172.21.15.78:0/392263252 conn(0x7fd0b40ff6e0 msgr2=0x7fd0b4101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:10.659 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0bb445700 1 -- 172.21.15.78:0/392263252 shutdown_connections 2023-07-14T18:56:10.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.073+0000 7fd0bb445700 1 -- 172.21.15.78:0/392263252 wait complete. 2023-07-14T18:56:10.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.074+0000 7fd0bb445700 1 Processor -- start 2023-07-14T18:56:10.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.074+0000 7fd0bb445700 1 -- start start 2023-07-14T18:56:10.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.074+0000 7fd0bb445700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b4198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:10.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.074+0000 7fd0bb445700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b41991b0 con 0x7fd0b4105cc0 2023-07-14T18:56:10.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.074+0000 7fd0b91e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b4198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:10.660 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0b91e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b4198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33272/0 (socket says 172.21.15.78:33272) 2023-07-14T18:56:10.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0b91e1700 1 -- 172.21.15.78:0/668830408 learned_addr learned my addr 172.21.15.78:0/668830408 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:10.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0b91e1700 1 -- 172.21.15.78:0/668830408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a4009550 con 0x7fd0b4105cc0 2023-07-14T18:56:10.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0b91e1700 1 --2- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b4198c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4000c00 tx=0x7fd0a4011e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:10.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0aa7fc700 1 -- 172.21.15.78:0/668830408 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0a40036a0 con 0x7fd0b4105cc0 2023-07-14T18:56:10.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0aa7fc700 1 -- 172.21.15.78:0/668830408 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fd0a401b440 con 0x7fd0b4105cc0 2023-07-14T18:56:10.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b41993b0 con 0x7fd0b4105cc0 2023-07-14T18:56:10.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0aa7fc700 1 -- 172.21.15.78:0/668830408 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd0a40193f0 con 0x7fd0b4105cc0 2023-07-14T18:56:10.661 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.075+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b4199830 con 0x7fd0b4105cc0 2023-07-14T18:56:10.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.076+0000 7fd0aa7fc700 1 -- 172.21.15.78:0/668830408 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 7) v1 ==== 45377+0+0 (secure 0 0 0) 0x7fd0a4019620 con 0x7fd0b4105cc0 2023-07-14T18:56:10.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.076+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b4109180 con 0x7fd0b4105cc0 2023-07-14T18:56:10.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.076+0000 7fd0aa7fc700 1 --2- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fd0a0038740 0x7fd0a003abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:10.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.076+0000 7fd0aa7fc700 1 -- 172.21.15.78:0/668830408 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd0a404c690 con 0x7fd0b4105cc0 2023-07-14T18:56:10.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.076+0000 7fd0b89e0700 1 --2- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fd0a0038740 0x7fd0a003abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:10.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.077+0000 7fd0b89e0700 1 --2- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fd0a0038740 0x7fd0a003abf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0006fd0 tx=0x7fd0b0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:10.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.081+0000 7fd0aa7fc700 1 -- 172.21.15.78:0/668830408 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd0a4022720 con 0x7fd0b4105cc0 2023-07-14T18:56:10.662 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.244+0000 7fd0aa7fc700 1 -- 172.21.15.78:0/668830408 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fd0a401b640 con 0x7fd0b4105cc0 2023-07-14T18:56:10.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.279+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fd0b4100a20 con 0x7fd0a0038740 2023-07-14T18:56:10.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.285+0000 7fd0aa7fc700 1 -- 172.21.15.78:0/668830408 <== mgr.14120 v2:172.21.15.78:6800/378103227 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd0b4100a20 con 0x7fd0a0038740 2023-07-14T18:56:10.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fd0a0038740 msgr2=0x7fd0a003abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:10.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 --2- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fd0a0038740 0x7fd0a003abf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd0b0006fd0 tx=0x7fd0b0006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:10.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 msgr2=0x7fd0b4198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:10.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 --2- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b4198c70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd0a4000c00 tx=0x7fd0a4011e70 comp rx=0 tx=0).stop 2023-07-14T18:56:10.663 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 shutdown_connections 2023-07-14T18:56:10.664 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 --2- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fd0a0038740 0x7fd0a003abf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:10.664 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 --2- 172.21.15.78:0/668830408 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0b4105cc0 0x7fd0b4198c70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:10.664 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 >> 172.21.15.78:0/668830408 conn(0x7fd0b40ff6e0 msgr2=0x7fd0b4100310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:10.664 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.287+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 shutdown_connections 2023-07-14T18:56:10.664 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:10.288+0000 7fd0bb445700 1 -- 172.21.15.78:0/668830408 wait complete. 2023-07-14T18:56:11.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:11 smithi078 ceph-mon[127425]: mgrmap e8: x(active, since 2s) 2023-07-14T18:56:11.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:11 smithi078 ceph-mon[127425]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:11.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:11 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:11.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:11 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:11.699 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout value unchanged 2023-07-14T18:56:11.699 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.170+0000 7f5883fd3700 1 Processor -- start 2023-07-14T18:56:11.700 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.170+0000 7f5883fd3700 1 -- start start 2023-07-14T18:56:11.700 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.170+0000 7f5883fd3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c079cd0 0x7f587c10a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:11.700 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.170+0000 7f5883fd3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587c105b10 con 0x7f587c079cd0 2023-07-14T18:56:11.700 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.171+0000 7f5881d6f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c079cd0 0x7f587c10a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:11.700 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.171+0000 7f5881d6f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c079cd0 0x7f587c10a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33276/0 (socket says 172.21.15.78:33276) 2023-07-14T18:56:11.700 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.171+0000 7f5881d6f700 1 -- 172.21.15.78:0/2633574248 learned_addr learned my addr 172.21.15.78:0/2633574248 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:11.700 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.171+0000 7f5881d6f700 1 -- 172.21.15.78:0/2633574248 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587c10a660 con 0x7f587c079cd0 2023-07-14T18:56:11.701 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.171+0000 7f5881d6f700 1 --2- 172.21.15.78:0/2633574248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c079cd0 0x7f587c10a120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f586c0098a0 tx=0x7f586c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6441087242537e2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:11.701 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.171+0000 7f5880d6d700 1 -- 172.21.15.78:0/2633574248 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f586c004160 con 0x7f587c079cd0 2023-07-14T18:56:11.701 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.171+0000 7f5880d6d700 1 -- 172.21.15.78:0/2633574248 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f586c0042c0 con 0x7f587c079cd0 2023-07-14T18:56:11.701 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.171+0000 7f5880d6d700 1 -- 172.21.15.78:0/2633574248 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f586c011940 con 0x7f587c079cd0 2023-07-14T18:56:11.701 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.172+0000 7f5883fd3700 1 -- 172.21.15.78:0/2633574248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c079cd0 msgr2=0x7f587c10a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:11.701 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.172+0000 7f5883fd3700 1 --2- 172.21.15.78:0/2633574248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c079cd0 0x7f587c10a120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f586c0098a0 tx=0x7f586c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:11.701 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.172+0000 7f5883fd3700 1 -- 172.21.15.78:0/2633574248 shutdown_connections 2023-07-14T18:56:11.702 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.172+0000 7f5883fd3700 1 --2- 172.21.15.78:0/2633574248 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c079cd0 0x7f587c10a120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:11.702 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.172+0000 7f5883fd3700 1 -- 172.21.15.78:0/2633574248 >> 172.21.15.78:0/2633574248 conn(0x7f587c0ff710 msgr2=0x7f587c101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:11.702 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.172+0000 7f5883fd3700 1 -- 172.21.15.78:0/2633574248 shutdown_connections 2023-07-14T18:56:11.702 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.172+0000 7f5883fd3700 1 -- 172.21.15.78:0/2633574248 wait complete. 2023-07-14T18:56:11.702 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5883fd3700 1 Processor -- start 2023-07-14T18:56:11.702 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5883fd3700 1 -- start start 2023-07-14T18:56:11.703 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5883fd3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c19b160 0x7f587c19b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:11.703 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5883fd3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f587c19bab0 con 0x7f587c19b160 2023-07-14T18:56:11.703 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5881d6f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c19b160 0x7f587c19b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:11.703 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5881d6f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c19b160 0x7f587c19b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33278/0 (socket says 172.21.15.78:33278) 2023-07-14T18:56:11.703 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5881d6f700 1 -- 172.21.15.78:0/4118371935 learned_addr learned my addr 172.21.15.78:0/4118371935 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:11.703 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5881d6f700 1 -- 172.21.15.78:0/4118371935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f586c009550 con 0x7f587c19b160 2023-07-14T18:56:11.703 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.173+0000 7f5881d6f700 1 --2- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c19b160 0x7f587c19b570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f586c000c00 tx=0x7f586c003a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:11.703 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.174+0000 7f5872ffd700 1 -- 172.21.15.78:0/4118371935 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f586c0118f0 con 0x7f587c19b160 2023-07-14T18:56:11.704 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.174+0000 7f5872ffd700 1 -- 172.21.15.78:0/4118371935 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f586c01b470 con 0x7f587c19b160 2023-07-14T18:56:11.704 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.174+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f587c19bcb0 con 0x7f587c19b160 2023-07-14T18:56:11.704 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.174+0000 7f5872ffd700 1 -- 172.21.15.78:0/4118371935 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f586c01c450 con 0x7f587c19b160 2023-07-14T18:56:11.704 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.174+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f587c19e910 con 0x7f587c19b160 2023-07-14T18:56:11.704 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.174+0000 7f5872ffd700 1 -- 172.21.15.78:0/4118371935 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f586c022070 con 0x7f587c19b160 2023-07-14T18:56:11.704 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.175+0000 7f5872ffd700 1 --2- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f5868038820 0x7f586803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:11.704 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.175+0000 7f5872ffd700 1 -- 172.21.15.78:0/4118371935 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f586c04c700 con 0x7f587c19b160 2023-07-14T18:56:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.175+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f587c061c70 con 0x7f587c19b160 2023-07-14T18:56:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.175+0000 7f588156e700 1 --2- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f5868038820 0x7f586803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.175+0000 7f588156e700 1 --2- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f5868038820 0x7f586803acd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5878006fd0 tx=0x7f5878006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.180+0000 7f5872ffd700 1 -- 172.21.15.78:0/4118371935 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f586c01c5b0 con 0x7f587c19b160 2023-07-14T18:56:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.365+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f587c101a20 con 0x7f5868038820 2023-07-14T18:56:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.366+0000 7f5872ffd700 1 -- 172.21.15.78:0/4118371935 <== mgr.14120 v2:172.21.15.78:6800/378103227 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f587c101a20 con 0x7f5868038820 2023-07-14T18:56:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f5868038820 msgr2=0x7f586803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:11.705 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 --2- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f5868038820 0x7f586803acd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5878006fd0 tx=0x7f5878006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c19b160 msgr2=0x7f587c19b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 --2- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c19b160 0x7f587c19b570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f586c000c00 tx=0x7f586c003a40 comp rx=0 tx=0).stop 2023-07-14T18:56:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 shutdown_connections 2023-07-14T18:56:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 --2- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f5868038820 0x7f586803acd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 --2- 172.21.15.78:0/4118371935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f587c19b160 0x7f587c19b570 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 >> 172.21.15.78:0/4118371935 conn(0x7f587c0ff710 msgr2=0x7f587c100d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:11.706 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 shutdown_connections 2023-07-14T18:56:11.707 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:11.368+0000 7f5883fd3700 1 -- 172.21.15.78:0/4118371935 wait complete. 2023-07-14T18:56:11.707 INFO:teuthology.orchestra.run.smithi078.stdout:Generating ssh key... 2023-07-14T18:56:12.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-mon[127425]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:12.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: Generating public/private rsa key pair. 2023-07-14T18:56:12.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: Your identification has been saved in /tmp/tmpp1jvmjwk/key. 2023-07-14T18:56:12.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: Your public key has been saved in /tmp/tmpp1jvmjwk/key.pub. 2023-07-14T18:56:12.550 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: The key fingerprint is: 2023-07-14T18:56:12.550 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: SHA256:y9YiBS5EbARvDtyMtK0hLwJQH58CNYBA2gMFBfQnQTQ ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:56:12.550 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: The key's randomart image is: 2023-07-14T18:56:12.550 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: +---[RSA 3072]----+ 2023-07-14T18:56:12.550 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: |XX@E* | 2023-07-14T18:56:12.551 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: |+=.@=+ . | 2023-07-14T18:56:12.551 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: |+ O+X.+ | 2023-07-14T18:56:12.551 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: |.o Ooo . | 2023-07-14T18:56:12.551 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: |o o o . S | 2023-07-14T18:56:12.551 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: |.. . o o | 2023-07-14T18:56:12.551 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: | . = . | 2023-07-14T18:56:12.551 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: | o . | 2023-07-14T18:56:12.552 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: | | 2023-07-14T18:56:12.552 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:12 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: +----[SHA256]-----+ 2023-07-14T18:56:12.843 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.143+0000 7f4dfee20700 1 Processor -- start 2023-07-14T18:56:12.843 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.143+0000 7f4dfee20700 1 -- start start 2023-07-14T18:56:12.843 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.143+0000 7f4dfee20700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df8104490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:12.843 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.143+0000 7f4dfee20700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df81049d0 con 0x7f4df8104080 2023-07-14T18:56:12.844 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.143+0000 7f4dfcbbc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df8104490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:12.844 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.143+0000 7f4dfcbbc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df8104490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33282/0 (socket says 172.21.15.78:33282) 2023-07-14T18:56:12.844 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.143+0000 7f4dfcbbc700 1 -- 172.21.15.78:0/3927453330 learned_addr learned my addr 172.21.15.78:0/3927453330 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:12.844 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.143+0000 7f4dfcbbc700 1 -- 172.21.15.78:0/3927453330 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df8104b10 con 0x7f4df8104080 2023-07-14T18:56:12.844 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4dfcbbc700 1 --2- 172.21.15.78:0/3927453330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df8104490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0098a0 tx=0x7f4dec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17ffcc00d555d58f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:12.845 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4df77fe700 1 -- 172.21.15.78:0/3927453330 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dec004160 con 0x7f4df8104080 2023-07-14T18:56:12.845 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4df77fe700 1 -- 172.21.15.78:0/3927453330 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4dec0042c0 con 0x7f4df8104080 2023-07-14T18:56:12.845 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4df77fe700 1 -- 172.21.15.78:0/3927453330 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dec011940 con 0x7f4df8104080 2023-07-14T18:56:12.845 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4dfee20700 1 -- 172.21.15.78:0/3927453330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 msgr2=0x7f4df8104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:12.845 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4dfee20700 1 --2- 172.21.15.78:0/3927453330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df8104490 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4dec0098a0 tx=0x7f4dec009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:12.845 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4dfee20700 1 -- 172.21.15.78:0/3927453330 shutdown_connections 2023-07-14T18:56:12.845 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4dfee20700 1 --2- 172.21.15.78:0/3927453330 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df8104490 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:12.846 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4dfee20700 1 -- 172.21.15.78:0/3927453330 >> 172.21.15.78:0/3927453330 conn(0x7f4df80ff710 msgr2=0x7f4df8101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:12.846 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4dfee20700 1 -- 172.21.15.78:0/3927453330 shutdown_connections 2023-07-14T18:56:12.846 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.144+0000 7f4dfee20700 1 -- 172.21.15.78:0/3927453330 wait complete. 2023-07-14T18:56:12.846 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.145+0000 7f4dfee20700 1 Processor -- start 2023-07-14T18:56:12.846 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.145+0000 7f4dfee20700 1 -- start start 2023-07-14T18:56:12.846 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.145+0000 7f4dfee20700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df818f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:12.846 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.145+0000 7f4dfee20700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df818f6e0 con 0x7f4df8104080 2023-07-14T18:56:12.847 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.145+0000 7f4dfcbbc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df818f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:12.847 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.145+0000 7f4dfcbbc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df818f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33284/0 (socket says 172.21.15.78:33284) 2023-07-14T18:56:12.847 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.145+0000 7f4dfcbbc700 1 -- 172.21.15.78:0/3346328512 learned_addr learned my addr 172.21.15.78:0/3346328512 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:12.847 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.145+0000 7f4dfcbbc700 1 -- 172.21.15.78:0/3346328512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dec009550 con 0x7f4df8104080 2023-07-14T18:56:12.847 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.146+0000 7f4dfcbbc700 1 --2- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df818f1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4dec011dc0 tx=0x7f4dec011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:12.847 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.146+0000 7f4df5ffb700 1 -- 172.21.15.78:0/3346328512 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dec0037d0 con 0x7f4df8104080 2023-07-14T18:56:12.847 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.146+0000 7f4df5ffb700 1 -- 172.21.15.78:0/3346328512 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f4dec01b440 con 0x7f4df8104080 2023-07-14T18:56:12.847 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.146+0000 7f4df5ffb700 1 -- 172.21.15.78:0/3346328512 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4dec01a560 con 0x7f4df8104080 2023-07-14T18:56:12.848 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.146+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df818d7f0 con 0x7f4df8104080 2023-07-14T18:56:12.848 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.146+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df818dc10 con 0x7f4df8104080 2023-07-14T18:56:12.848 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.147+0000 7f4df5ffb700 1 -- 172.21.15.78:0/3346328512 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f4dec01b5b0 con 0x7f4df8104080 2023-07-14T18:56:12.848 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.147+0000 7f4df5ffb700 1 --2- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f4de8038820 0x7f4de803acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:12.848 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.147+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df8061c70 con 0x7f4df8104080 2023-07-14T18:56:12.848 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.147+0000 7f4df5ffb700 1 -- 172.21.15.78:0/3346328512 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f4dec04c5d0 con 0x7f4df8104080 2023-07-14T18:56:12.848 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.147+0000 7f4df7fff700 1 --2- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f4de8038820 0x7f4de803acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:12.849 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.148+0000 7f4df7fff700 1 --2- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f4de8038820 0x7f4de803acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4de4006fd0 tx=0x7f4de4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:12.849 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.152+0000 7f4df5ffb700 1 -- 172.21.15.78:0/3346328512 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4dec01f020 con 0x7f4df8104080 2023-07-14T18:56:12.849 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.338+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4df818e6a0 con 0x7f4de8038820 2023-07-14T18:56:12.849 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.412+0000 7f4df5ffb700 1 -- 172.21.15.78:0/3346328512 <== mgr.14120 v2:172.21.15.78:6800/378103227 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f4df818e6a0 con 0x7f4de8038820 2023-07-14T18:56:12.849 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f4de8038820 msgr2=0x7f4de803acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 --2- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f4de8038820 0x7f4de803acd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4de4006fd0 tx=0x7f4de4006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 msgr2=0x7f4df818f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 --2- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df818f1a0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4dec011dc0 tx=0x7f4dec011ea0 comp rx=0 tx=0).stop 2023-07-14T18:56:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 shutdown_connections 2023-07-14T18:56:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 --2- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f4de8038820 0x7f4de803acd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 --2- 172.21.15.78:0/3346328512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4df8104080 0x7f4df818f1a0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 >> 172.21.15.78:0/3346328512 conn(0x7f4df80ff710 msgr2=0x7f4df8079930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:12.850 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 shutdown_connections 2023-07-14T18:56:12.851 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:12.414+0000 7f4dfee20700 1 -- 172.21.15.78:0/3346328512 wait complete. 2023-07-14T18:56:13.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:13 smithi078 ceph-mon[127425]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:13.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:13 smithi078 ceph-mon[127425]: Generating ssh key... 2023-07-14T18:56:13.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:13 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:13.690 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:13 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:13.885 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr3JZMYQ+W65+lp46KnvX8Bg7p0lyisZP7Ia/muor73+TjY8g5Muc6Dasm2Bd6O8pOWlP42+hDUw7bHs+lVzD+rYtjyEYicc49nbXIK807sKbsByWNOjRkwpL+NS86147vgON/TA+pXMW9c6TTbHW3hD/LvON7iY1zsK8Qom5kgIkSYvQF+9rt0X17vC9wDtFGgDd1fyKOMvuG2Tv7GEB1c7P3AFW32ZOuhAfaLBaEaWrzQuaJ8sNMCNgyy6HTZi7XsLn6NCEbR46xVXx8fYr7HWOzXDg4uWYsm2txloNXm5p8wumuson4cfBWnCdP4nNdil4plhVabHvH5CZAbs7nJSOpCQXvQDdCYR9Zgmo57AAIf1Z16QcaSzSpS9jE3Ty4VHlBSzf43z7FntKoS3Zr/b8LvCnk0DGipRfW4IHdT70q+WfAMOcEpEq5kfwW49Ogg5VZsypU/qDlHo2hnDUfxUJQ5imBtNyWurysK05SQudIPEMusbUuKTqBjm22gU= ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:56:13.885 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.320+0000 7fc371e08700 1 Processor -- start 2023-07-14T18:56:13.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.320+0000 7fc371e08700 1 -- start start 2023-07-14T18:56:13.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.320+0000 7fc371e08700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c103c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:13.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.320+0000 7fc371e08700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c104160 con 0x7fc36c103810 2023-07-14T18:56:13.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.320+0000 7fc36b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c103c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:13.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.320+0000 7fc36b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c103c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33288/0 (socket says 172.21.15.78:33288) 2023-07-14T18:56:13.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.320+0000 7fc36b7fe700 1 -- 172.21.15.78:0/3876249930 learned_addr learned my addr 172.21.15.78:0/3876249930 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.320+0000 7fc36b7fe700 1 -- 172.21.15.78:0/3876249930 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c1042a0 con 0x7fc36c103810 2023-07-14T18:56:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc36b7fe700 1 --2- 172.21.15.78:0/3876249930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c103c20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0098a0 tx=0x7fc35c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8302aa703bfee64d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc36a7fc700 1 -- 172.21.15.78:0/3876249930 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc35c004160 con 0x7fc36c103810 2023-07-14T18:56:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc36a7fc700 1 -- 172.21.15.78:0/3876249930 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc35c0042c0 con 0x7fc36c103810 2023-07-14T18:56:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc36a7fc700 1 -- 172.21.15.78:0/3876249930 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc35c011940 con 0x7fc36c103810 2023-07-14T18:56:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc371e08700 1 -- 172.21.15.78:0/3876249930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 msgr2=0x7fc36c103c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc371e08700 1 --2- 172.21.15.78:0/3876249930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c103c20 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc35c0098a0 tx=0x7fc35c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:13.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc371e08700 1 -- 172.21.15.78:0/3876249930 shutdown_connections 2023-07-14T18:56:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc371e08700 1 --2- 172.21.15.78:0/3876249930 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c103c20 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc371e08700 1 -- 172.21.15.78:0/3876249930 >> 172.21.15.78:0/3876249930 conn(0x7fc36c0fee80 msgr2=0x7fc36c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc371e08700 1 -- 172.21.15.78:0/3876249930 shutdown_connections 2023-07-14T18:56:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.321+0000 7fc371e08700 1 -- 172.21.15.78:0/3876249930 wait complete. 2023-07-14T18:56:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.322+0000 7fc371e08700 1 Processor -- start 2023-07-14T18:56:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.322+0000 7fc371e08700 1 -- start start 2023-07-14T18:56:13.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.322+0000 7fc371e08700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c196b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:13.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.322+0000 7fc371e08700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc36c197070 con 0x7fc36c103810 2023-07-14T18:56:13.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.322+0000 7fc36b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c196b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:13.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.322+0000 7fc36b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c196b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33290/0 (socket says 172.21.15.78:33290) 2023-07-14T18:56:13.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.322+0000 7fc36b7fe700 1 -- 172.21.15.78:0/2670833370 learned_addr learned my addr 172.21.15.78:0/2670833370 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:13.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.323+0000 7fc36b7fe700 1 -- 172.21.15.78:0/2670833370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c009550 con 0x7fc36c103810 2023-07-14T18:56:13.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.323+0000 7fc36b7fe700 1 --2- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c196b30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc35c011dc0 tx=0x7fc35c011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:13.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.323+0000 7fc368ff9700 1 -- 172.21.15.78:0/2670833370 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc35c0037d0 con 0x7fc36c103810 2023-07-14T18:56:13.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.323+0000 7fc368ff9700 1 -- 172.21.15.78:0/2670833370 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fc35c01b440 con 0x7fc36c103810 2023-07-14T18:56:13.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.323+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc36c197270 con 0x7fc36c103810 2023-07-14T18:56:13.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.323+0000 7fc368ff9700 1 -- 172.21.15.78:0/2670833370 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc35c01a560 con 0x7fc36c103810 2023-07-14T18:56:13.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.323+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc36c197690 con 0x7fc36c103810 2023-07-14T18:56:13.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.324+0000 7fc368ff9700 1 -- 172.21.15.78:0/2670833370 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fc35c01a7d0 con 0x7fc36c103810 2023-07-14T18:56:13.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.324+0000 7fc368ff9700 1 --2- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc35403aa70 0x7fc35403cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:13.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.324+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc36c190340 con 0x7fc36c103810 2023-07-14T18:56:13.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.324+0000 7fc368ff9700 1 -- 172.21.15.78:0/2670833370 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc35c04c7a0 con 0x7fc36c103810 2023-07-14T18:56:13.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.324+0000 7fc36affd700 1 --2- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc35403aa70 0x7fc35403cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:13.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.325+0000 7fc36affd700 1 --2- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc35403aa70 0x7fc35403cf20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc360006fd0 tx=0x7fc360006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:13.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.329+0000 7fc368ff9700 1 -- 172.21.15.78:0/2670833370 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc35c0239a0 con 0x7fc36c103810 2023-07-14T18:56:13.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.514+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fc36c060a80 con 0x7fc35403aa70 2023-07-14T18:56:13.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.515+0000 7fc368ff9700 1 -- 172.21.15.78:0/2670833370 <== mgr.14120 v2:172.21.15.78:6800/378103227 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fc36c060a80 con 0x7fc35403aa70 2023-07-14T18:56:13.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc35403aa70 msgr2=0x7fc35403cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:13.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 --2- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc35403aa70 0x7fc35403cf20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc360006fd0 tx=0x7fc360006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:13.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 msgr2=0x7fc36c196b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:13.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 --2- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c196b30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc35c011dc0 tx=0x7fc35c011ea0 comp rx=0 tx=0).stop 2023-07-14T18:56:13.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 shutdown_connections 2023-07-14T18:56:13.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 --2- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fc35403aa70 0x7fc35403cf20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:13.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 --2- 172.21.15.78:0/2670833370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc36c103810 0x7fc36c196b30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:13.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 >> 172.21.15.78:0/2670833370 conn(0x7fc36c0fee80 msgr2=0x7fc36c18b270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:13.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 shutdown_connections 2023-07-14T18:56:13.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:13.517+0000 7fc371e08700 1 -- 172.21.15.78:0/2670833370 wait complete. 2023-07-14T18:56:13.893 INFO:teuthology.orchestra.run.smithi078.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-07-14T18:56:13.893 INFO:teuthology.orchestra.run.smithi078.stdout:Adding key to root@localhost authorized_keys... 2023-07-14T18:56:13.893 INFO:teuthology.orchestra.run.smithi078.stdout:Adding host smithi078... 2023-07-14T18:56:14.665 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:14 smithi078 ceph-mon[127425]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:15.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:15 smithi078 ceph-mon[127425]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:16.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:16 smithi078 ceph-mon[127425]: Deploying cephadm binary to smithi078 2023-07-14T18:56:17.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Added host 'smithi078' with addr '172.21.15.78' 2023-07-14T18:56:17.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0beaa2700 1 Processor -- start 2023-07-14T18:56:17.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0beaa2700 1 -- start start 2023-07-14T18:56:17.236 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0beaa2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b81080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:17.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0beaa2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8079cf0 con 0x7fa0b8105ce0 2023-07-14T18:56:17.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0bc83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b81080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:17.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0bc83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b81080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33294/0 (socket says 172.21.15.78:33294) 2023-07-14T18:56:17.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0bc83e700 1 -- 172.21.15.78:0/2890086097 learned_addr learned my addr 172.21.15.78:0/2890086097 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:17.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0bc83e700 1 -- 172.21.15.78:0/2890086097 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8079e30 con 0x7fa0b8105ce0 2023-07-14T18:56:17.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.388+0000 7fa0bc83e700 1 --2- 172.21.15.78:0/2890086097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b81080c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0098a0 tx=0x7fa0ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c058e20f15656ef server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:17.237 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0b77fe700 1 -- 172.21.15.78:0/2890086097 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ac004160 con 0x7fa0b8105ce0 2023-07-14T18:56:17.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0b77fe700 1 -- 172.21.15.78:0/2890086097 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa0ac0042c0 con 0x7fa0b8105ce0 2023-07-14T18:56:17.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0b77fe700 1 -- 172.21.15.78:0/2890086097 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ac011940 con 0x7fa0b8105ce0 2023-07-14T18:56:17.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0beaa2700 1 -- 172.21.15.78:0/2890086097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 msgr2=0x7fa0b81080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:17.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0beaa2700 1 --2- 172.21.15.78:0/2890086097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b81080c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac0098a0 tx=0x7fa0ac009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:17.238 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0beaa2700 1 -- 172.21.15.78:0/2890086097 shutdown_connections 2023-07-14T18:56:17.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0beaa2700 1 --2- 172.21.15.78:0/2890086097 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b81080c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:17.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0beaa2700 1 -- 172.21.15.78:0/2890086097 >> 172.21.15.78:0/2890086097 conn(0x7fa0b80ff6e0 msgr2=0x7fa0b8101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:17.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0beaa2700 1 -- 172.21.15.78:0/2890086097 shutdown_connections 2023-07-14T18:56:17.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.389+0000 7fa0beaa2700 1 -- 172.21.15.78:0/2890086097 wait complete. 2023-07-14T18:56:17.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0beaa2700 1 Processor -- start 2023-07-14T18:56:17.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0beaa2700 1 -- start start 2023-07-14T18:56:17.239 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0beaa2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b819aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:17.240 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0beaa2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b819b3e0 con 0x7fa0b8105ce0 2023-07-14T18:56:17.240 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0bc83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b819aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:17.240 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0bc83e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b819aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33296/0 (socket says 172.21.15.78:33296) 2023-07-14T18:56:17.240 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0bc83e700 1 -- 172.21.15.78:0/3018043234 learned_addr learned my addr 172.21.15.78:0/3018043234 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:17.240 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0bc83e700 1 -- 172.21.15.78:0/3018043234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac009550 con 0x7fa0b8105ce0 2023-07-14T18:56:17.240 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.390+0000 7fa0bc83e700 1 --2- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b819aea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac006af0 tx=0x7fa0ac011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:17.240 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.391+0000 7fa0b5ffb700 1 -- 172.21.15.78:0/3018043234 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ac003bf0 con 0x7fa0b8105ce0 2023-07-14T18:56:17.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.391+0000 7fa0b5ffb700 1 -- 172.21.15.78:0/3018043234 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7fa0ac01b8b0 con 0x7fa0b8105ce0 2023-07-14T18:56:17.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.391+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b819b5e0 con 0x7fa0b8105ce0 2023-07-14T18:56:17.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.391+0000 7fa0b5ffb700 1 -- 172.21.15.78:0/3018043234 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa0ac0198b0 con 0x7fa0b8105ce0 2023-07-14T18:56:17.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.391+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b819ba00 con 0x7fa0b8105ce0 2023-07-14T18:56:17.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.392+0000 7fa0b5ffb700 1 -- 172.21.15.78:0/3018043234 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fa0ac01ba20 con 0x7fa0b8105ce0 2023-07-14T18:56:17.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.392+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b8109210 con 0x7fa0b8105ce0 2023-07-14T18:56:17.241 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.392+0000 7fa0b5ffb700 1 --2- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fa0a8038780 0x7fa0a803ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:17.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.392+0000 7fa0b5ffb700 1 -- 172.21.15.78:0/3018043234 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fa0ac04c9e0 con 0x7fa0b8105ce0 2023-07-14T18:56:17.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.392+0000 7fa0b7fff700 1 --2- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fa0a8038780 0x7fa0a803ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:17.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.392+0000 7fa0b7fff700 1 --2- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fa0a8038780 0x7fa0a803ac30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4006fd0 tx=0x7fa0a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:17.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.396+0000 7fa0b5ffb700 1 -- 172.21.15.78:0/3018043234 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa0ac018920 con 0x7fa0b8105ce0 2023-07-14T18:56:17.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:14.585+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi078", "addr": "172.21.15.78", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0b819e860 con 0x7fa0a8038780 2023-07-14T18:56:17.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.849+0000 7fa0b5ffb700 1 -- 172.21.15.78:0/3018043234 <== mgr.14120 v2:172.21.15.78:6800/378103227 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fa0b819e860 con 0x7fa0a8038780 2023-07-14T18:56:17.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.851+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fa0a8038780 msgr2=0x7fa0a803ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:17.242 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.851+0000 7fa0beaa2700 1 --2- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fa0a8038780 0x7fa0a803ac30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa0a4006fd0 tx=0x7fa0a4006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:17.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.852+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 msgr2=0x7fa0b819aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:17.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.852+0000 7fa0beaa2700 1 --2- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b819aea0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac006af0 tx=0x7fa0ac011a30 comp rx=0 tx=0).stop 2023-07-14T18:56:17.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.852+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 shutdown_connections 2023-07-14T18:56:17.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.852+0000 7fa0beaa2700 1 --2- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fa0a8038780 0x7fa0a803ac30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:17.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.852+0000 7fa0beaa2700 1 --2- 172.21.15.78:0/3018043234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa0b8105ce0 0x7fa0b819aea0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:17.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.852+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 >> 172.21.15.78:0/3018043234 conn(0x7fa0b80ff6e0 msgr2=0x7fa0b8100350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:17.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.852+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 shutdown_connections 2023-07-14T18:56:17.243 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:16.852+0000 7fa0beaa2700 1 -- 172.21.15.78:0/3018043234 wait complete. 2023-07-14T18:56:17.244 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mon service... 2023-07-14T18:56:18.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:17 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:18.131 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:17 smithi078 ceph-mon[127425]: Added host smithi078 2023-07-14T18:56:18.132 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:17 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:18.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-07-14T18:56:18.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.723+0000 7f160f270700 1 Processor -- start 2023-07-14T18:56:18.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.723+0000 7f160f270700 1 -- start start 2023-07-14T18:56:18.383 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.723+0000 7f160f270700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f16081080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:18.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.723+0000 7f160f270700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1608079d10 con 0x7f1608105cc0 2023-07-14T18:56:18.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.723+0000 7f160d00c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f16081080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:18.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.724+0000 7f160d00c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f16081080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33330/0 (socket says 172.21.15.78:33330) 2023-07-14T18:56:18.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.724+0000 7f160d00c700 1 -- 172.21.15.78:0/291980660 learned_addr learned my addr 172.21.15.78:0/291980660 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:18.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.724+0000 7f160d00c700 1 -- 172.21.15.78:0/291980660 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1608079e50 con 0x7f1608105cc0 2023-07-14T18:56:18.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.724+0000 7f160d00c700 1 --2- 172.21.15.78:0/291980660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f16081080a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f15f80098a0 tx=0x7f15f8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b19ec3d2961c15bb server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:18.384 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.724+0000 7f15fffff700 1 -- 172.21.15.78:0/291980660 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15f8004160 con 0x7f1608105cc0 2023-07-14T18:56:18.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.724+0000 7f15fffff700 1 -- 172.21.15.78:0/291980660 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f15f80042c0 con 0x7f1608105cc0 2023-07-14T18:56:18.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.724+0000 7f15fffff700 1 -- 172.21.15.78:0/291980660 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15f8011940 con 0x7f1608105cc0 2023-07-14T18:56:18.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.725+0000 7f160f270700 1 -- 172.21.15.78:0/291980660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 msgr2=0x7f16081080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:18.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.725+0000 7f160f270700 1 --2- 172.21.15.78:0/291980660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f16081080a0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f15f80098a0 tx=0x7f15f8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:18.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.725+0000 7f160f270700 1 -- 172.21.15.78:0/291980660 shutdown_connections 2023-07-14T18:56:18.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.725+0000 7f160f270700 1 --2- 172.21.15.78:0/291980660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f16081080a0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:18.385 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.725+0000 7f160f270700 1 -- 172.21.15.78:0/291980660 >> 172.21.15.78:0/291980660 conn(0x7f16080ff6e0 msgr2=0x7f1608101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:18.386 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.725+0000 7f160f270700 1 -- 172.21.15.78:0/291980660 shutdown_connections 2023-07-14T18:56:18.386 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.725+0000 7f160f270700 1 -- 172.21.15.78:0/291980660 wait complete. 2023-07-14T18:56:18.386 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160f270700 1 Processor -- start 2023-07-14T18:56:18.386 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160f270700 1 -- start start 2023-07-14T18:56:18.386 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160f270700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f1608198c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:18.386 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160f270700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16081991b0 con 0x7f1608105cc0 2023-07-14T18:56:18.386 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160d00c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f1608198c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:18.387 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160d00c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f1608198c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33332/0 (socket says 172.21.15.78:33332) 2023-07-14T18:56:18.387 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160d00c700 1 -- 172.21.15.78:0/205179057 learned_addr learned my addr 172.21.15.78:0/205179057 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:18.387 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160d00c700 1 -- 172.21.15.78:0/205179057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f8009550 con 0x7f1608105cc0 2023-07-14T18:56:18.387 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160d00c700 1 --2- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f1608198c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f15f8006af0 tx=0x7f15f8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:18.387 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f15fe7fc700 1 -- 172.21.15.78:0/205179057 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15f8003bf0 con 0x7f1608105cc0 2023-07-14T18:56:18.387 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f15fe7fc700 1 -- 172.21.15.78:0/205179057 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f15f801b8b0 con 0x7f1608105cc0 2023-07-14T18:56:18.387 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1608199410 con 0x7f1608105cc0 2023-07-14T18:56:18.387 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.726+0000 7f15fe7fc700 1 -- 172.21.15.78:0/205179057 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15f80198b0 con 0x7f1608105cc0 2023-07-14T18:56:18.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.727+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1608199890 con 0x7f1608105cc0 2023-07-14T18:56:18.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.727+0000 7f15fe7fc700 1 -- 172.21.15.78:0/205179057 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f15f8019b70 con 0x7f1608105cc0 2023-07-14T18:56:18.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.727+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1608109180 con 0x7f1608105cc0 2023-07-14T18:56:18.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.727+0000 7f15fe7fc700 1 --2- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f15f4038780 0x7f15f403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:18.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.728+0000 7f15fe7fc700 1 -- 172.21.15.78:0/205179057 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f15f804cbd0 con 0x7f1608105cc0 2023-07-14T18:56:18.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.728+0000 7f160c80b700 1 --2- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f15f4038780 0x7f15f403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:18.388 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.728+0000 7f160c80b700 1 --2- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f15f4038780 0x7f15f403ac30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1604006fd0 tx=0x7f1604006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:18.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.732+0000 7f15fe7fc700 1 -- 172.21.15.78:0/205179057 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f15f8018b00 con 0x7f1608105cc0 2023-07-14T18:56:18.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.942+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1608100a20 con 0x7f15f4038780 2023-07-14T18:56:18.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.947+0000 7f15fe7fc700 1 -- 172.21.15.78:0/205179057 <== mgr.14120 v2:172.21.15.78:6800/378103227 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1608100a20 con 0x7f15f4038780 2023-07-14T18:56:18.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.949+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f15f4038780 msgr2=0x7f15f403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:18.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.949+0000 7f160f270700 1 --2- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f15f4038780 0x7f15f403ac30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1604006fd0 tx=0x7f1604006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:18.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.949+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 msgr2=0x7f1608198c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:18.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.949+0000 7f160f270700 1 --2- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f1608198c70 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f15f8006af0 tx=0x7f15f8011a30 comp rx=0 tx=0).stop 2023-07-14T18:56:18.389 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.949+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 shutdown_connections 2023-07-14T18:56:18.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.949+0000 7f160f270700 1 --2- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f15f4038780 0x7f15f403ac30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:18.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.949+0000 7f160f270700 1 --2- 172.21.15.78:0/205179057 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1608105cc0 0x7f1608198c70 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:18.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.949+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 >> 172.21.15.78:0/205179057 conn(0x7f16080ff6e0 msgr2=0x7f1608100310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:18.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.950+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 shutdown_connections 2023-07-14T18:56:18.390 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:17.950+0000 7f160f270700 1 -- 172.21.15.78:0/205179057 wait complete. 2023-07-14T18:56:18.390 INFO:teuthology.orchestra.run.smithi078.stdout:Deploying unmanaged mgr service... 2023-07-14T18:56:19.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:18 smithi078 ceph-mon[127425]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:19.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:18 smithi078 ceph-mon[127425]: Saving service mon spec with placement count:5 2023-07-14T18:56:19.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:18 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:19.676 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-07-14T18:56:19.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.042+0000 7f82b0a2b700 1 Processor -- start 2023-07-14T18:56:19.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.042+0000 7f82b0a2b700 1 -- start start 2023-07-14T18:56:19.677 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.042+0000 7f82b0a2b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac104480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.042+0000 7f82b0a2b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac1049c0 con 0x7f82ac104070 2023-07-14T18:56:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.042+0000 7f82aa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac104480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.043+0000 7f82aa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac104480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33336/0 (socket says 172.21.15.78:33336) 2023-07-14T18:56:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.043+0000 7f82aa59c700 1 -- 172.21.15.78:0/1447752765 learned_addr learned my addr 172.21.15.78:0/1447752765 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:19.678 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.043+0000 7f82aa59c700 1 -- 172.21.15.78:0/1447752765 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac104b00 con 0x7f82ac104070 2023-07-14T18:56:19.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.043+0000 7f82aa59c700 1 --2- 172.21.15.78:0/1447752765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac104480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f82a00098a0 tx=0x7f82a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b249b9b3435fb5b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:19.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.043+0000 7f82a959a700 1 -- 172.21.15.78:0/1447752765 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82a0004160 con 0x7f82ac104070 2023-07-14T18:56:19.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.043+0000 7f82a959a700 1 -- 172.21.15.78:0/1447752765 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f82a00042c0 con 0x7f82ac104070 2023-07-14T18:56:19.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.043+0000 7f82a959a700 1 -- 172.21.15.78:0/1447752765 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82a0011940 con 0x7f82ac104070 2023-07-14T18:56:19.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1447752765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 msgr2=0x7f82ac104480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:19.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 --2- 172.21.15.78:0/1447752765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac104480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f82a00098a0 tx=0x7f82a0009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:19.679 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1447752765 shutdown_connections 2023-07-14T18:56:19.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 --2- 172.21.15.78:0/1447752765 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac104480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:19.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1447752765 >> 172.21.15.78:0/1447752765 conn(0x7f82ac0ff6e0 msgr2=0x7f82ac101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:19.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1447752765 shutdown_connections 2023-07-14T18:56:19.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1447752765 wait complete. 2023-07-14T18:56:19.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 Processor -- start 2023-07-14T18:56:19.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.044+0000 7f82b0a2b700 1 -- start start 2023-07-14T18:56:19.680 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82b0a2b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac079b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:19.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82b0a2b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac1997c0 con 0x7f82ac104070 2023-07-14T18:56:19.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82aa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac079b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:19.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82aa59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac079b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33338/0 (socket says 172.21.15.78:33338) 2023-07-14T18:56:19.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82aa59c700 1 -- 172.21.15.78:0/1360873349 learned_addr learned my addr 172.21.15.78:0/1360873349 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:19.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82aa59c700 1 -- 172.21.15.78:0/1360873349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a0009550 con 0x7f82ac104070 2023-07-14T18:56:19.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82aa59c700 1 --2- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac079b00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f82a0011dc0 tx=0x7f82a0011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:19.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f829b7fe700 1 -- 172.21.15.78:0/1360873349 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82a00037d0 con 0x7f82ac104070 2023-07-14T18:56:19.681 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac19ddd0 con 0x7f82ac104070 2023-07-14T18:56:19.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac19e130 con 0x7f82ac104070 2023-07-14T18:56:19.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f829b7fe700 1 -- 172.21.15.78:0/1360873349 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1036+0+0 (secure 0 0 0) 0x7f82a001b440 con 0x7f82ac104070 2023-07-14T18:56:19.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.045+0000 7f829b7fe700 1 -- 172.21.15.78:0/1360873349 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f82a001a560 con 0x7f82ac104070 2023-07-14T18:56:19.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.046+0000 7f829b7fe700 1 -- 172.21.15.78:0/1360873349 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f82a001a7d0 con 0x7f82ac104070 2023-07-14T18:56:19.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.046+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f828c005320 con 0x7f82ac104070 2023-07-14T18:56:19.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.046+0000 7f829b7fe700 1 --2- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f8294038820 0x7f829403acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:19.682 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.046+0000 7f829b7fe700 1 -- 172.21.15.78:0/1360873349 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f82a004c610 con 0x7f82ac104070 2023-07-14T18:56:19.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.047+0000 7f82a9d9b700 1 --2- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f8294038820 0x7f829403acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:19.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.047+0000 7f82a9d9b700 1 --2- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f8294038820 0x7f829403acd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f829c006fd0 tx=0x7f829c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:19.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.051+0000 7f829b7fe700 1 -- 172.21.15.78:0/1360873349 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f82a0023590 con 0x7f82ac104070 2023-07-14T18:56:19.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.178+0000 7f829b7fe700 1 -- 172.21.15.78:0/1360873349 <== mon.0 v2:172.21.15.78:3300/0 7 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f82a004c080 con 0x7f82ac104070 2023-07-14T18:56:19.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.259+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f828c000bf0 con 0x7f8294038820 2023-07-14T18:56:19.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.263+0000 7f829b7fe700 1 -- 172.21.15.78:0/1360873349 <== mgr.14120 v2:172.21.15.78:6800/378103227 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f828c000bf0 con 0x7f8294038820 2023-07-14T18:56:19.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f8294038820 msgr2=0x7f829403acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:19.683 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 --2- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f8294038820 0x7f829403acd0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f829c006fd0 tx=0x7f829c006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:19.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 msgr2=0x7f82ac079b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:19.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 --2- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac079b00 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f82a0011dc0 tx=0x7f82a0011ea0 comp rx=0 tx=0).stop 2023-07-14T18:56:19.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 shutdown_connections 2023-07-14T18:56:19.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 --2- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f8294038820 0x7f829403acd0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:19.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 --2- 172.21.15.78:0/1360873349 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f82ac104070 0x7f82ac079b00 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:19.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 >> 172.21.15.78:0/1360873349 conn(0x7f82ac0ff6e0 msgr2=0x7f82ac100310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:19.684 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.265+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 shutdown_connections 2023-07-14T18:56:19.685 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:19.266+0000 7f82b0a2b700 1 -- 172.21.15.78:0/1360873349 wait complete. 2023-07-14T18:56:20.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:20 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:20.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:20 smithi078 ceph-mon[127425]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:20.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:20 smithi078 ceph-mon[127425]: Saving service mgr spec with placement count:2 2023-07-14T18:56:20.184 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:20 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:20.185 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:20 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:20.720 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.128+0000 7fef94d2e700 1 Processor -- start 2023-07-14T18:56:20.720 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.128+0000 7fef94d2e700 1 -- start start 2023-07-14T18:56:20.720 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.128+0000 7fef94d2e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef90105a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:20.721 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.128+0000 7fef94d2e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef90105f40 con 0x7fef901055f0 2023-07-14T18:56:20.721 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.128+0000 7fef8e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef90105a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:20.721 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.128+0000 7fef8e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef90105a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33342/0 (socket says 172.21.15.78:33342) 2023-07-14T18:56:20.721 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.128+0000 7fef8e59c700 1 -- 172.21.15.78:0/261548453 learned_addr learned my addr 172.21.15.78:0/261548453 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:20.721 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.129+0000 7fef8e59c700 1 -- 172.21.15.78:0/261548453 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90106080 con 0x7fef901055f0 2023-07-14T18:56:20.721 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.129+0000 7fef8e59c700 1 --2- 172.21.15.78:0/261548453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef90105a00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef800098a0 tx=0x7fef80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5e0b4568f733af server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:20.721 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.129+0000 7fef8d59a700 1 -- 172.21.15.78:0/261548453 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef80004160 con 0x7fef901055f0 2023-07-14T18:56:20.722 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.129+0000 7fef8d59a700 1 -- 172.21.15.78:0/261548453 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fef800042c0 con 0x7fef901055f0 2023-07-14T18:56:20.722 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.129+0000 7fef8d59a700 1 -- 172.21.15.78:0/261548453 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef800118e0 con 0x7fef901055f0 2023-07-14T18:56:20.722 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.130+0000 7fef94d2e700 1 -- 172.21.15.78:0/261548453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 msgr2=0x7fef90105a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:20.722 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.130+0000 7fef94d2e700 1 --2- 172.21.15.78:0/261548453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef90105a00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fef800098a0 tx=0x7fef80009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:20.722 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.130+0000 7fef94d2e700 1 -- 172.21.15.78:0/261548453 shutdown_connections 2023-07-14T18:56:20.722 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.130+0000 7fef94d2e700 1 --2- 172.21.15.78:0/261548453 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef90105a00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:20.722 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.130+0000 7fef94d2e700 1 -- 172.21.15.78:0/261548453 >> 172.21.15.78:0/261548453 conn(0x7fef900fee20 msgr2=0x7fef90101230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:20.723 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.130+0000 7fef94d2e700 1 -- 172.21.15.78:0/261548453 shutdown_connections 2023-07-14T18:56:20.723 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.130+0000 7fef94d2e700 1 -- 172.21.15.78:0/261548453 wait complete. 2023-07-14T18:56:20.723 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.131+0000 7fef94d2e700 1 Processor -- start 2023-07-14T18:56:20.723 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.131+0000 7fef94d2e700 1 -- start start 2023-07-14T18:56:20.723 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.131+0000 7fef94d2e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef9018f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:20.723 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.131+0000 7fef94d2e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9018f690 con 0x7fef901055f0 2023-07-14T18:56:20.723 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.131+0000 7fef8e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef9018f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:20.724 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.131+0000 7fef8e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef9018f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33344/0 (socket says 172.21.15.78:33344) 2023-07-14T18:56:20.724 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.131+0000 7fef8e59c700 1 -- 172.21.15.78:0/640443206 learned_addr learned my addr 172.21.15.78:0/640443206 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:20.724 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.131+0000 7fef8e59c700 1 -- 172.21.15.78:0/640443206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef80009550 con 0x7fef901055f0 2023-07-14T18:56:20.724 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.132+0000 7fef8e59c700 1 --2- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef9018f150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fef80006af0 tx=0x7fef80011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:20.724 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.132+0000 7fef7f7fe700 1 -- 172.21.15.78:0/640443206 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef800037d0 con 0x7fef901055f0 2023-07-14T18:56:20.724 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.132+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef9018d7a0 con 0x7fef901055f0 2023-07-14T18:56:20.724 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.132+0000 7fef7f7fe700 1 -- 172.21.15.78:0/640443206 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fef8001b440 con 0x7fef901055f0 2023-07-14T18:56:20.725 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.132+0000 7fef7f7fe700 1 -- 172.21.15.78:0/640443206 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef800193d0 con 0x7fef901055f0 2023-07-14T18:56:20.725 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.132+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9018dbc0 con 0x7fef901055f0 2023-07-14T18:56:20.725 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.133+0000 7fef7f7fe700 1 -- 172.21.15.78:0/640443206 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7fef80019640 con 0x7fef901055f0 2023-07-14T18:56:20.725 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.133+0000 7fef7f7fe700 1 --2- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fef780387d0 0x7fef7803ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:20.725 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.133+0000 7fef7f7fe700 1 -- 172.21.15.78:0/640443206 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fef8004d410 con 0x7fef901055f0 2023-07-14T18:56:20.725 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.133+0000 7fef8dd9b700 1 --2- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fef780387d0 0x7fef7803ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:20.725 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.133+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef90104ed0 con 0x7fef901055f0 2023-07-14T18:56:20.726 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.133+0000 7fef8dd9b700 1 --2- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fef780387d0 0x7fef7803ac80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fef84006fd0 tx=0x7fef84006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:20.726 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.138+0000 7fef7f7fe700 1 -- 172.21.15.78:0/640443206 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fef80018920 con 0x7fef901055f0 2023-07-14T18:56:20.726 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.358+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fef90105090 con 0x7fef901055f0 2023-07-14T18:56:20.726 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.362+0000 7fef7f7fe700 1 -- 172.21.15.78:0/640443206 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fef8001f210 con 0x7fef901055f0 2023-07-14T18:56:20.726 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fef780387d0 msgr2=0x7fef7803ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:20.726 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 --2- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fef780387d0 0x7fef7803ac80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fef84006fd0 tx=0x7fef84006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:20.726 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 msgr2=0x7fef9018f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:20.727 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 --2- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef9018f150 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fef80006af0 tx=0x7fef80011e80 comp rx=0 tx=0).stop 2023-07-14T18:56:20.727 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 shutdown_connections 2023-07-14T18:56:20.727 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 --2- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fef780387d0 0x7fef7803ac80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:20.727 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 --2- 172.21.15.78:0/640443206 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef901055f0 0x7fef9018f150 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:20.727 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 >> 172.21.15.78:0/640443206 conn(0x7fef900fee20 msgr2=0x7fef90103b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:20.727 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 shutdown_connections 2023-07-14T18:56:20.727 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:20.364+0000 7fef94d2e700 1 -- 172.21.15.78:0/640443206 wait complete. 2023-07-14T18:56:21.740 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:21 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/640443206' entity='client.admin' 2023-07-14T18:56:21.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.167+0000 7f2ce0108700 1 Processor -- start 2023-07-14T18:56:21.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.168+0000 7f2ce0108700 1 -- start start 2023-07-14T18:56:21.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.168+0000 7f2ce0108700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8106b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:21.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.168+0000 7f2ce0108700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81070c0 con 0x7f2cd8106770 2023-07-14T18:56:21.766 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.168+0000 7f2cddea4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8106b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:21.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.168+0000 7f2cddea4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8106b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33348/0 (socket says 172.21.15.78:33348) 2023-07-14T18:56:21.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.168+0000 7f2cddea4700 1 -- 172.21.15.78:0/1310010894 learned_addr learned my addr 172.21.15.78:0/1310010894 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:21.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.168+0000 7f2cddea4700 1 -- 172.21.15.78:0/1310010894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd8107200 con 0x7f2cd8106770 2023-07-14T18:56:21.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2cddea4700 1 --2- 172.21.15.78:0/1310010894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8106b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40098a0 tx=0x7f2cd4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8363cdfa0d4412e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:21.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2cdcea2700 1 -- 172.21.15.78:0/1310010894 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cd4004160 con 0x7f2cd8106770 2023-07-14T18:56:21.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2cdcea2700 1 -- 172.21.15.78:0/1310010894 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2cd40042c0 con 0x7f2cd8106770 2023-07-14T18:56:21.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2cdcea2700 1 -- 172.21.15.78:0/1310010894 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cd40118e0 con 0x7f2cd8106770 2023-07-14T18:56:21.767 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2ce0108700 1 -- 172.21.15.78:0/1310010894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 msgr2=0x7f2cd8106b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:21.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2ce0108700 1 --2- 172.21.15.78:0/1310010894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8106b80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2cd40098a0 tx=0x7f2cd4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:21.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2ce0108700 1 -- 172.21.15.78:0/1310010894 shutdown_connections 2023-07-14T18:56:21.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2ce0108700 1 --2- 172.21.15.78:0/1310010894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8106b80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:21.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.169+0000 7f2ce0108700 1 -- 172.21.15.78:0/1310010894 >> 172.21.15.78:0/1310010894 conn(0x7f2cd80fee80 msgr2=0x7f2cd8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:21.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.170+0000 7f2ce0108700 1 -- 172.21.15.78:0/1310010894 shutdown_connections 2023-07-14T18:56:21.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.170+0000 7f2ce0108700 1 -- 172.21.15.78:0/1310010894 wait complete. 2023-07-14T18:56:21.768 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.170+0000 7f2ce0108700 1 Processor -- start 2023-07-14T18:56:21.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.170+0000 7f2ce0108700 1 -- start start 2023-07-14T18:56:21.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2ce0108700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:21.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2ce0108700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cd81992d0 con 0x7f2cd8106770 2023-07-14T18:56:21.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2cddea4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:21.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2cddea4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33350/0 (socket says 172.21.15.78:33350) 2023-07-14T18:56:21.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2cddea4700 1 -- 172.21.15.78:0/3099804196 learned_addr learned my addr 172.21.15.78:0/3099804196 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:21.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2cddea4700 1 -- 172.21.15.78:0/3099804196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd4009550 con 0x7f2cd8106770 2023-07-14T18:56:21.769 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2cddea4700 1 --2- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8198d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4000c00 tx=0x7f2cd40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:21.770 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2cceffd700 1 -- 172.21.15.78:0/3099804196 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cd40036a0 con 0x7f2cd8106770 2023-07-14T18:56:21.770 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2cceffd700 1 -- 172.21.15.78:0/3099804196 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2cd401b440 con 0x7f2cd8106770 2023-07-14T18:56:21.770 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cd81994d0 con 0x7f2cd8106770 2023-07-14T18:56:21.770 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2cceffd700 1 -- 172.21.15.78:0/3099804196 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2cd401a3d0 con 0x7f2cd8106770 2023-07-14T18:56:21.770 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.171+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cd81998f0 con 0x7f2cd8106770 2023-07-14T18:56:21.770 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.172+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cd8061c70 con 0x7f2cd8106770 2023-07-14T18:56:21.771 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.173+0000 7f2cceffd700 1 -- 172.21.15.78:0/3099804196 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7f2cd401b5b0 con 0x7f2cd8106770 2023-07-14T18:56:21.771 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.173+0000 7f2cceffd700 1 --2- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f2cc40387d0 0x7f2cc403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:21.771 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.173+0000 7f2cceffd700 1 -- 172.21.15.78:0/3099804196 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2cd404c720 con 0x7f2cd8106770 2023-07-14T18:56:21.771 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.173+0000 7f2cdd6a3700 1 --2- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f2cc40387d0 0x7f2cc403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:21.771 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.173+0000 7f2cdd6a3700 1 --2- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f2cc40387d0 0x7f2cc403ac80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8006fd0 tx=0x7f2cc8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:21.771 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.182+0000 7f2cceffd700 1 -- 172.21.15.78:0/3099804196 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2cd40039e0 con 0x7f2cd8106770 2023-07-14T18:56:21.771 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.375+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f2cd819c2f0 con 0x7f2cd8106770 2023-07-14T18:56:21.771 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.381+0000 7f2cceffd700 1 -- 172.21.15.78:0/3099804196 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f2cd401f020 con 0x7f2cd8106770 2023-07-14T18:56:21.772 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f2cc40387d0 msgr2=0x7f2cc403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:21.772 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 --2- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f2cc40387d0 0x7f2cc403ac80 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f2cc8006fd0 tx=0x7f2cc8006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:21.772 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 msgr2=0x7f2cd8198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:21.772 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 --2- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8198d90 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4000c00 tx=0x7f2cd40119a0 comp rx=0 tx=0).stop 2023-07-14T18:56:21.772 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 shutdown_connections 2023-07-14T18:56:21.772 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 --2- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f2cc40387d0 0x7f2cc403ac80 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:21.773 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 --2- 172.21.15.78:0/3099804196 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2cd8106770 0x7f2cd8198d90 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:21.773 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 >> 172.21.15.78:0/3099804196 conn(0x7f2cd80fee80 msgr2=0x7f2cd80ffb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:21.773 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 shutdown_connections 2023-07-14T18:56:21.773 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:21.383+0000 7f2ce0108700 1 -- 172.21.15.78:0/3099804196 wait complete. 2023-07-14T18:56:21.773 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling the dashboard module... 2023-07-14T18:56:22.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:22 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3099804196' entity='client.admin' 2023-07-14T18:56:22.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:22 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:23.663 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:23 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3401469944' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-07-14T18:56:23.663 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:23 smithi078 ceph-mon[127425]: from='mgr.14120 172.21.15.78:0/2124955082' entity='mgr.x' 2023-07-14T18:56:23.664 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:23 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: ignoring --setuser ceph since I am not root 2023-07-14T18:56:23.664 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:23 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: ignoring --setgroup ceph since I am not root 2023-07-14T18:56:23.664 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:23 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:23.575+0000 7ff35c176200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:56:23.664 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:23 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:23.657+0000 7ff35c176200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:56:24.028 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:23 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:23.752+0000 7ff35c176200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:56:24.028 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:23 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:23.851+0000 7ff35c176200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:56:24.029 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:24 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:24.022+0000 7ff35c176200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:56:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.311+0000 7efd4e2a5700 1 Processor -- start 2023-07-14T18:56:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.311+0000 7efd4e2a5700 1 -- start start 2023-07-14T18:56:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.311+0000 7efd4e2a5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd48079cd0 0x7efd4810a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:24.214 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.311+0000 7efd4e2a5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd48105b00 con 0x7efd48079cd0 2023-07-14T18:56:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.312+0000 7efd47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd48079cd0 0x7efd4810a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.312+0000 7efd47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd48079cd0 0x7efd4810a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33354/0 (socket says 172.21.15.78:33354) 2023-07-14T18:56:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.312+0000 7efd47fff700 1 -- 172.21.15.78:0/3136668508 learned_addr learned my addr 172.21.15.78:0/3136668508 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.312+0000 7efd47fff700 1 -- 172.21.15.78:0/3136668508 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd4810a650 con 0x7efd48079cd0 2023-07-14T18:56:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.312+0000 7efd47fff700 1 --2- 172.21.15.78:0/3136668508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd48079cd0 0x7efd4810a110 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd300098a0 tx=0x7efd30009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b0053c678f82ed2 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.312+0000 7efd46ffd700 1 -- 172.21.15.78:0/3136668508 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd30004160 con 0x7efd48079cd0 2023-07-14T18:56:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.312+0000 7efd46ffd700 1 -- 172.21.15.78:0/3136668508 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efd300042c0 con 0x7efd48079cd0 2023-07-14T18:56:24.215 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.312+0000 7efd46ffd700 1 -- 172.21.15.78:0/3136668508 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd300118e0 con 0x7efd48079cd0 2023-07-14T18:56:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3136668508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd48079cd0 msgr2=0x7efd4810a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 --2- 172.21.15.78:0/3136668508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd48079cd0 0x7efd4810a110 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efd300098a0 tx=0x7efd30009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3136668508 shutdown_connections 2023-07-14T18:56:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 --2- 172.21.15.78:0/3136668508 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd48079cd0 0x7efd4810a110 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3136668508 >> 172.21.15.78:0/3136668508 conn(0x7efd480ff700 msgr2=0x7efd48101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:24.216 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3136668508 shutdown_connections 2023-07-14T18:56:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3136668508 wait complete. 2023-07-14T18:56:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 Processor -- start 2023-07-14T18:56:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.313+0000 7efd4e2a5700 1 -- start start 2023-07-14T18:56:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd4e2a5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd4819af10 0x7efd4819b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd4e2a5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd4819b860 con 0x7efd4819af10 2023-07-14T18:56:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd4819af10 0x7efd4819b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd4819af10 0x7efd4819b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33356/0 (socket says 172.21.15.78:33356) 2023-07-14T18:56:24.217 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd47fff700 1 -- 172.21.15.78:0/3401469944 learned_addr learned my addr 172.21.15.78:0/3401469944 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd47fff700 1 -- 172.21.15.78:0/3401469944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd30009550 con 0x7efd4819af10 2023-07-14T18:56:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd47fff700 1 --2- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd4819af10 0x7efd4819b320 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd30006af0 tx=0x7efd30011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd457fa700 1 -- 172.21.15.78:0/3401469944 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd300037d0 con 0x7efd4819af10 2023-07-14T18:56:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd457fa700 1 -- 172.21.15.78:0/3401469944 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efd3001b440 con 0x7efd4819af10 2023-07-14T18:56:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd4819ba60 con 0x7efd4819af10 2023-07-14T18:56:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd457fa700 1 -- 172.21.15.78:0/3401469944 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efd30019560 con 0x7efd4819af10 2023-07-14T18:56:24.218 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.314+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd4819e6c0 con 0x7efd4819af10 2023-07-14T18:56:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.315+0000 7efd457fa700 1 -- 172.21.15.78:0/3401469944 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 8) v1 ==== 45483+0+0 (secure 0 0 0) 0x7efd30019820 con 0x7efd4819af10 2023-07-14T18:56:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.315+0000 7efd457fa700 1 --2- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7efd34038780 0x7efd3403ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.315+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd48079cd0 con 0x7efd4819af10 2023-07-14T18:56:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.315+0000 7efd457fa700 1 -- 172.21.15.78:0/3401469944 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7efd3004c740 con 0x7efd4819af10 2023-07-14T18:56:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.316+0000 7efd477fe700 1 --2- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7efd34038780 0x7efd3403ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.316+0000 7efd477fe700 1 --2- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7efd34038780 0x7efd3403ac30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd38006fd0 tx=0x7efd38006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.320+0000 7efd457fa700 1 -- 172.21.15.78:0/3401469944 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efd30018350 con 0x7efd4819af10 2023-07-14T18:56:24.219 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:22.573+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7efd48079ec0 con 0x7efd4819af10 2023-07-14T18:56:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.385+0000 7efd457fa700 1 -- 172.21.15.78:0/3401469944 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7efd3001a8f0 con 0x7efd4819af10 2023-07-14T18:56:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.385+0000 7efd457fa700 1 -- 172.21.15.78:0/3401469944 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7efd3001b820 con 0x7efd4819af10 2023-07-14T18:56:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.388+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7efd34038780 msgr2=0x7efd3403ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.388+0000 7efd4e2a5700 1 --2- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7efd34038780 0x7efd3403ac30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7efd38006fd0 tx=0x7efd38006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.388+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd4819af10 msgr2=0x7efd4819b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.388+0000 7efd4e2a5700 1 --2- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd4819af10 0x7efd4819b320 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efd30006af0 tx=0x7efd30011e80 comp rx=0 tx=0).stop 2023-07-14T18:56:24.220 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.388+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 shutdown_connections 2023-07-14T18:56:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.388+0000 7efd4e2a5700 1 --2- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7efd34038780 0x7efd3403ac30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.388+0000 7efd4e2a5700 1 --2- 172.21.15.78:0/3401469944 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd4819af10 0x7efd4819b320 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.388+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 >> 172.21.15.78:0/3401469944 conn(0x7efd480ff700 msgr2=0x7efd481003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.389+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 shutdown_connections 2023-07-14T18:56:24.221 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:23.389+0000 7efd4e2a5700 1 -- 172.21.15.78:0/3401469944 wait complete. 2023-07-14T18:56:24.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:24 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3401469944' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-07-14T18:56:24.612 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:24 smithi078 ceph-mon[127425]: mgrmap e9: x(active, since 15s) 2023-07-14T18:56:24.613 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:24 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:24.340+0000 7ff35c176200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:56:24.613 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:24 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:24.434+0000 7ff35c176200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:56:24.613 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:24 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:24.566+0000 7ff35c176200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:56:25.049 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:24 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:24.666+0000 7ff35c176200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:56:25.389 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:25.103+0000 7ff35c176200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:56:25.572 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:56:25.572 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-07-14T18:56:25.572 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "available": true, 2023-07-14T18:56:25.573 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-07-14T18:56:25.573 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-07-14T18:56:25.573 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:56:25.573 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.865+0000 7f34fd8fe700 1 Processor -- start 2023-07-14T18:56:25.573 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.865+0000 7f34fd8fe700 1 -- start start 2023-07-14T18:56:25.573 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.865+0000 7f34fd8fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8108070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:25.573 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.865+0000 7f34fd8fe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f8079d50 con 0x7f34f8105c90 2023-07-14T18:56:25.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.866+0000 7f34f6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8108070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:25.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.866+0000 7f34f6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8108070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33372/0 (socket says 172.21.15.78:33372) 2023-07-14T18:56:25.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.866+0000 7f34f6ffd700 1 -- 172.21.15.78:0/4194722403 learned_addr learned my addr 172.21.15.78:0/4194722403 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:25.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.866+0000 7f34f6ffd700 1 -- 172.21.15.78:0/4194722403 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34f8079e90 con 0x7f34f8105c90 2023-07-14T18:56:25.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.866+0000 7f34f6ffd700 1 --2- 172.21.15.78:0/4194722403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8108070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f34e80098a0 tx=0x7f34e8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac2dd16337938c4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:25.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.866+0000 7f34f5ffb700 1 -- 172.21.15.78:0/4194722403 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34e8004160 con 0x7f34f8105c90 2023-07-14T18:56:25.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.866+0000 7f34f5ffb700 1 -- 172.21.15.78:0/4194722403 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f34e80042c0 con 0x7f34f8105c90 2023-07-14T18:56:25.574 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.866+0000 7f34f5ffb700 1 -- 172.21.15.78:0/4194722403 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34e80118e0 con 0x7f34f8105c90 2023-07-14T18:56:25.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.867+0000 7f34fd8fe700 1 -- 172.21.15.78:0/4194722403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 msgr2=0x7f34f8108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:25.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.867+0000 7f34fd8fe700 1 --2- 172.21.15.78:0/4194722403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8108070 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f34e80098a0 tx=0x7f34e8009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:25.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.867+0000 7f34fd8fe700 1 -- 172.21.15.78:0/4194722403 shutdown_connections 2023-07-14T18:56:25.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.867+0000 7f34fd8fe700 1 --2- 172.21.15.78:0/4194722403 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8108070 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:25.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.867+0000 7f34fd8fe700 1 -- 172.21.15.78:0/4194722403 >> 172.21.15.78:0/4194722403 conn(0x7f34f80ff710 msgr2=0x7f34f8101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:25.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.867+0000 7f34fd8fe700 1 -- 172.21.15.78:0/4194722403 shutdown_connections 2023-07-14T18:56:25.575 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.867+0000 7f34fd8fe700 1 -- 172.21.15.78:0/4194722403 wait complete. 2023-07-14T18:56:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.867+0000 7f34fd8fe700 1 Processor -- start 2023-07-14T18:56:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34fd8fe700 1 -- start start 2023-07-14T18:56:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34fd8fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8198db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34fd8fe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34f81992f0 con 0x7f34f8105c90 2023-07-14T18:56:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34f6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8198db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34f6ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8198db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33374/0 (socket says 172.21.15.78:33374) 2023-07-14T18:56:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34f6ffd700 1 -- 172.21.15.78:0/3897935647 learned_addr learned my addr 172.21.15.78:0/3897935647 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:25.576 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34f6ffd700 1 -- 172.21.15.78:0/3897935647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34e8009550 con 0x7f34f8105c90 2023-07-14T18:56:25.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34f6ffd700 1 --2- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8198db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f34e8000c00 tx=0x7f34e80119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:25.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34fc8fc700 1 -- 172.21.15.78:0/3897935647 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34e80036a0 con 0x7f34f8105c90 2023-07-14T18:56:25.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34fc8fc700 1 -- 172.21.15.78:0/3897935647 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f34e801b440 con 0x7f34f8105c90 2023-07-14T18:56:25.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34fc8fc700 1 -- 172.21.15.78:0/3897935647 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f34e801a3d0 con 0x7f34f8105c90 2023-07-14T18:56:25.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.868+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34f81994f0 con 0x7f34f8105c90 2023-07-14T18:56:25.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.869+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34f8199910 con 0x7f34f8105c90 2023-07-14T18:56:25.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.870+0000 7f34fc8fc700 1 -- 172.21.15.78:0/3897935647 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f34e801a640 con 0x7f34f8105c90 2023-07-14T18:56:25.577 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.870+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34f8061c70 con 0x7f34f8105c90 2023-07-14T18:56:25.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.870+0000 7f34fc8fc700 1 --2- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f34e0038870 0x7f34e003ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:25.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.870+0000 7f34fc8fc700 1 -- 172.21.15.78:0/3897935647 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f34e804c7d0 con 0x7f34f8105c90 2023-07-14T18:56:25.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.870+0000 7f34f67fc700 1 -- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f34e0038870 msgr2=0x7f34e003ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/378103227 2023-07-14T18:56:25.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.870+0000 7f34f67fc700 1 --2- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f34e0038870 0x7f34e003ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:56:25.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:24.874+0000 7f34fc8fc700 1 -- 172.21.15.78:0/3897935647 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f34e8018d20 con 0x7f34f8105c90 2023-07-14T18:56:25.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.071+0000 7f34f67fc700 1 -- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f34e0038870 msgr2=0x7f34e003ad20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/378103227 2023-07-14T18:56:25.578 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.071+0000 7f34f67fc700 1 --2- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f34e0038870 0x7f34e003ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:56:25.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.120+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f34f819c440 con 0x7f34f8105c90 2023-07-14T18:56:25.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.120+0000 7f34fc8fc700 1 -- 172.21.15.78:0/3897935647 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f34e8018b40 con 0x7f34f8105c90 2023-07-14T18:56:25.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.122+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f34e0038870 msgr2=0x7f34e003ad20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:56:25.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.122+0000 7f34fd8fe700 1 --2- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f34e0038870 0x7f34e003ad20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:25.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.122+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 msgr2=0x7f34f8198db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:25.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.122+0000 7f34fd8fe700 1 --2- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8198db0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f34e8000c00 tx=0x7f34e80119a0 comp rx=0 tx=0).stop 2023-07-14T18:56:25.579 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.122+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 shutdown_connections 2023-07-14T18:56:25.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.122+0000 7f34fd8fe700 1 --2- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7f34e0038870 0x7f34e003ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:25.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.122+0000 7f34fd8fe700 1 --2- 172.21.15.78:0/3897935647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34f8105c90 0x7f34f8198db0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:25.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.122+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 >> 172.21.15.78:0/3897935647 conn(0x7f34f80ff710 msgr2=0x7f34f8100430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:25.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.123+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 shutdown_connections 2023-07-14T18:56:25.580 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:25.123+0000 7f34fd8fe700 1 -- 172.21.15.78:0/3897935647 wait complete. 2023-07-14T18:56:25.580 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for the mgr to restart... 2023-07-14T18:56:25.580 INFO:teuthology.orchestra.run.smithi078.stdout:Waiting for mgr epoch 9... 2023-07-14T18:56:25.704 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:25 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3897935647' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-07-14T18:56:26.049 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:25.699+0000 7ff35c176200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:56:26.049 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:25.880+0000 7ff35c176200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:56:26.921 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:26 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:26.651+0000 7ff35c176200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:56:26.922 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:26 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:26.743+0000 7ff35c176200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:56:27.197 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:26 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:26.916+0000 7ff35c176200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:56:27.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:27 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:27.192+0000 7ff35c176200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:56:27.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:27 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:27.417+0000 7ff35c176200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:56:27.832 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:27 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:27.566+0000 7ff35c176200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:56:28.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:27 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:27.827+0000 7ff35c176200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:56:28.799 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:28 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:28.521+0000 7ff35c176200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:56:28.799 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:28 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:28.651+0000 7ff35c176200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:56:29.177 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:28 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:28.842+0000 7ff35c176200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:56:29.177 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:28 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:28.956+0000 7ff35c176200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:56:29.549 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:29 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:29.172+0000 7ff35c176200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:56:30.299 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T18:56:30.022+0000 7ff35c176200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:56:30.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: Active manager daemon x restarted 2023-07-14T18:56:30.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: Activating manager daemon x 2023-07-14T18:56:30.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: osdmap e3: 0 total, 0 up, 0 in 2023-07-14T18:56:30.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: mgrmap e10: x(active, starting, since 0.00336708s) 2023-07-14T18:56:30.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:56:30.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-07-14T18:56:30.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-07-14T18:56:30.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-07-14T18:56:30.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-07-14T18:56:30.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: Manager daemon x is now available 2023-07-14T18:56:30.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:30.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:31.410 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-07-14T18:56:31.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-07-14T18:56:31.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:31 smithi078 ceph-mon[127425]: [14/Jul/2023:18:56:30] ENGINE Bus STARTING 2023-07-14T18:56:31.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:31 smithi078 ceph-mon[127425]: mgrmap e11: x(active, since 1.00552s) 2023-07-14T18:56:32.045 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout { 2023-07-14T18:56:32.045 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-07-14T18:56:32.046 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout "initialized": true 2023-07-14T18:56:32.046 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout } 2023-07-14T18:56:32.046 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.071+0000 7fcdc969d700 1 Processor -- start 2023-07-14T18:56:32.046 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.071+0000 7fcdc969d700 1 -- start start 2023-07-14T18:56:32.046 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc969d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc41080d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:32.046 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc969d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc4079cf0 con 0x7fcdc4105cf0 2023-07-14T18:56:32.046 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc41080d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:32.047 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc41080d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33380/0 (socket says 172.21.15.78:33380) 2023-07-14T18:56:32.047 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc2ffd700 1 -- 172.21.15.78:0/3568371079 learned_addr learned my addr 172.21.15.78:0/3568371079 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:32.047 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc2ffd700 1 -- 172.21.15.78:0/3568371079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc4079e30 con 0x7fcdc4105cf0 2023-07-14T18:56:32.047 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc2ffd700 1 --2- 172.21.15.78:0/3568371079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc41080d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40098a0 tx=0x7fcdb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d52507d58fb9ef56 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:32.047 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc1ffb700 1 -- 172.21.15.78:0/3568371079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb4004160 con 0x7fcdc4105cf0 2023-07-14T18:56:32.047 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc1ffb700 1 -- 172.21.15.78:0/3568371079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcdb40042c0 con 0x7fcdc4105cf0 2023-07-14T18:56:32.047 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.072+0000 7fcdc1ffb700 1 -- 172.21.15.78:0/3568371079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb40118e0 con 0x7fcdc4105cf0 2023-07-14T18:56:32.048 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.073+0000 7fcdc969d700 1 -- 172.21.15.78:0/3568371079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 msgr2=0x7fcdc41080d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:32.048 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.073+0000 7fcdc969d700 1 --2- 172.21.15.78:0/3568371079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc41080d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcdb40098a0 tx=0x7fcdb4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:32.048 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.073+0000 7fcdc969d700 1 -- 172.21.15.78:0/3568371079 shutdown_connections 2023-07-14T18:56:32.048 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.073+0000 7fcdc969d700 1 --2- 172.21.15.78:0/3568371079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc41080d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:32.048 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.073+0000 7fcdc969d700 1 -- 172.21.15.78:0/3568371079 >> 172.21.15.78:0/3568371079 conn(0x7fcdc40ff710 msgr2=0x7fcdc4101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:32.048 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.073+0000 7fcdc969d700 1 -- 172.21.15.78:0/3568371079 shutdown_connections 2023-07-14T18:56:32.048 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.073+0000 7fcdc969d700 1 -- 172.21.15.78:0/3568371079 wait complete. 2023-07-14T18:56:32.048 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.074+0000 7fcdc969d700 1 Processor -- start 2023-07-14T18:56:32.049 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.074+0000 7fcdc969d700 1 -- start start 2023-07-14T18:56:32.049 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.074+0000 7fcdc969d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc419afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:32.049 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.074+0000 7fcdc969d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc419b4f0 con 0x7fcdc4105cf0 2023-07-14T18:56:32.049 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.074+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc419afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:32.049 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.074+0000 7fcdc2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc419afb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33382/0 (socket says 172.21.15.78:33382) 2023-07-14T18:56:32.049 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.074+0000 7fcdc2ffd700 1 -- 172.21.15.78:0/1779840656 learned_addr learned my addr 172.21.15.78:0/1779840656 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:32.049 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.074+0000 7fcdc2ffd700 1 -- 172.21.15.78:0/1779840656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdb4009550 con 0x7fcdc4105cf0 2023-07-14T18:56:32.050 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.075+0000 7fcdc2ffd700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc419afb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4006af0 tx=0x7fcdb4011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:32.050 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.075+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb40037d0 con 0x7fcdc4105cf0 2023-07-14T18:56:32.050 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.075+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc419b6f0 con 0x7fcdc4105cf0 2023-07-14T18:56:32.050 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.075+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcdb401b440 con 0x7fcdc4105cf0 2023-07-14T18:56:32.050 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.075+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcdb4019560 con 0x7fcdc4105cf0 2023-07-14T18:56:32.050 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.075+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc419bb10 con 0x7fcdc4105cf0 2023-07-14T18:56:32.050 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.076+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 9) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fcdb4019820 con 0x7fcdc4105cf0 2023-07-14T18:56:32.050 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.076+0000 7fcdabfff700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 0x7fcdac03ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:32.051 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.076+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 --> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcdac03b390 con 0x7fcdac0387d0 2023-07-14T18:56:32.051 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.076+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcdb404c5b0 con 0x7fcdc4105cf0 2023-07-14T18:56:32.051 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.076+0000 7fcdc27fc700 1 -- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 msgr2=0x7fcdac03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/378103227 2023-07-14T18:56:32.051 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.076+0000 7fcdc27fc700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 0x7fcdac03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:56:32.051 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.277+0000 7fcdc27fc700 1 -- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 msgr2=0x7fcdac03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/378103227 2023-07-14T18:56:32.051 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.277+0000 7fcdc27fc700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 0x7fcdac03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-07-14T18:56:32.051 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.678+0000 7fcdc27fc700 1 -- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 msgr2=0x7fcdac03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/378103227 2023-07-14T18:56:32.051 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:26.678+0000 7fcdc27fc700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 0x7fcdac03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-07-14T18:56:32.052 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:27.479+0000 7fcdc27fc700 1 -- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 msgr2=0x7fcdac03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/378103227 2023-07-14T18:56:32.052 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:27.479+0000 7fcdc27fc700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 0x7fcdac03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-07-14T18:56:32.052 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:29.080+0000 7fcdc27fc700 1 -- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 msgr2=0x7fcdac03ac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.78:6800/378103227 2023-07-14T18:56:32.052 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:29.080+0000 7fcdc27fc700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 0x7fcdac03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-07-14T18:56:32.052 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:30.038+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mgrmap(e 10) v1 ==== 45263+0+0 (secure 0 0 0) 0x7fcdb4019b70 con 0x7fcdc4105cf0 2023-07-14T18:56:32.052 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:30.038+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 msgr2=0x7fcdac03ac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:56:32.052 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:30.038+0000 7fcdabfff700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 0x7fcdac03ac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:32.053 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.039+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 11) v1 ==== 45390+0+0 (secure 0 0 0) 0x7fcdb4022b80 con 0x7fcdc4105cf0 2023-07-14T18:56:32.053 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.040+0000 7fcdabfff700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdac03bfe0 0x7fcdac03e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:32.053 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.040+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcdac03b390 con 0x7fcdac03bfe0 2023-07-14T18:56:32.053 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.040+0000 7fcdc27fc700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdac03bfe0 0x7fcdac03e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:32.053 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.040+0000 7fcdc27fc700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdac03bfe0 0x7fcdac03e3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8003a10 tx=0x7fcdb80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:32.053 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.041+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6910 (secure 0 0 0) 0x7fcdac03b390 con 0x7fcdac03bfe0 2023-07-14T18:56:32.053 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.047+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fcdc419c2f0 con 0x7fcdac03bfe0 2023-07-14T18:56:32.053 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.047+0000 7fcdabfff700 1 -- 172.21.15.78:0/1779840656 <== mgr.14152 v2:172.21.15.78:6800/2266091126 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fcdc419c2f0 con 0x7fcdac03bfe0 2023-07-14T18:56:32.054 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.047+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdac03bfe0 msgr2=0x7fcdac03e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:32.054 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.047+0000 7fcdc969d700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdac03bfe0 0x7fcdac03e3c0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8003a10 tx=0x7fcdb80092b0 comp rx=0 tx=0).stop 2023-07-14T18:56:32.054 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.047+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 msgr2=0x7fcdc419afb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:32.054 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.047+0000 7fcdc969d700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc419afb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4006af0 tx=0x7fcdb4011e80 comp rx=0 tx=0).stop 2023-07-14T18:56:32.054 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.048+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 shutdown_connections 2023-07-14T18:56:32.054 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.048+0000 7fcdc969d700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdac03bfe0 0x7fcdac03e3c0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:32.054 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.048+0000 7fcdc969d700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcdc4105cf0 0x7fcdc419afb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:32.054 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.048+0000 7fcdc969d700 1 --2- 172.21.15.78:0/1779840656 >> [v2:172.21.15.78:6800/378103227,v1:172.21.15.78:6801/378103227] conn(0x7fcdac0387d0 0x7fcdac03ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:32.055 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.048+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 >> 172.21.15.78:0/1779840656 conn(0x7fcdc40ff710 msgr2=0x7fcdc4100430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:32.055 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.048+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 shutdown_connections 2023-07-14T18:56:32.055 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:31.048+0000 7fcdc969d700 1 -- 172.21.15.78:0/1779840656 wait complete. 2023-07-14T18:56:32.055 INFO:teuthology.orchestra.run.smithi078.stdout:mgr epoch 9 is available 2023-07-14T18:56:32.055 INFO:teuthology.orchestra.run.smithi078.stdout:Generating a dashboard self-signed certificate... 2023-07-14T18:56:32.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:32 smithi078 ceph-mon[127425]: [14/Jul/2023:18:56:30] ENGINE Serving on https://172.21.15.78:7150 2023-07-14T18:56:32.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:32 smithi078 ceph-mon[127425]: [14/Jul/2023:18:56:30] ENGINE Serving on http://172.21.15.78:8765 2023-07-14T18:56:32.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:32 smithi078 ceph-mon[127425]: [14/Jul/2023:18:56:30] ENGINE Bus STARTED 2023-07-14T18:56:32.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:32 smithi078 ceph-mon[127425]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-07-14T18:56:32.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:32 smithi078 ceph-mon[127425]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-07-14T18:56:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:33 smithi078 ceph-mon[127425]: mgrmap e12: x(active, since 2s) 2023-07-14T18:56:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:33.339 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-07-14T18:56:33.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.596+0000 7f0181a73700 1 Processor -- start 2023-07-14T18:56:33.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.596+0000 7f0181a73700 1 -- start start 2023-07-14T18:56:33.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.596+0000 7f0181a73700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c104490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:33.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.596+0000 7f0181a73700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c1049d0 con 0x7f017c104080 2023-07-14T18:56:33.340 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.597+0000 7f017b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c104490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.597+0000 7f017b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c104490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33436/0 (socket says 172.21.15.78:33436) 2023-07-14T18:56:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.597+0000 7f017b7fe700 1 -- 172.21.15.78:0/2782360442 learned_addr learned my addr 172.21.15.78:0/2782360442 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.597+0000 7f017b7fe700 1 -- 172.21.15.78:0/2782360442 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c104b10 con 0x7f017c104080 2023-07-14T18:56:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.597+0000 7f017b7fe700 1 --2- 172.21.15.78:0/2782360442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c104490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f016c0098a0 tx=0x7f016c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=462f1734627d9f9e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.597+0000 7f017a7fc700 1 -- 172.21.15.78:0/2782360442 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f016c004160 con 0x7f017c104080 2023-07-14T18:56:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.597+0000 7f017a7fc700 1 -- 172.21.15.78:0/2782360442 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f016c0042c0 con 0x7f017c104080 2023-07-14T18:56:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.597+0000 7f017a7fc700 1 -- 172.21.15.78:0/2782360442 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f016c0118e0 con 0x7f017c104080 2023-07-14T18:56:33.341 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.598+0000 7f0181a73700 1 -- 172.21.15.78:0/2782360442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 msgr2=0x7f017c104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.598+0000 7f0181a73700 1 --2- 172.21.15.78:0/2782360442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c104490 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f016c0098a0 tx=0x7f016c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.598+0000 7f0181a73700 1 -- 172.21.15.78:0/2782360442 shutdown_connections 2023-07-14T18:56:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.598+0000 7f0181a73700 1 --2- 172.21.15.78:0/2782360442 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c104490 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.598+0000 7f0181a73700 1 -- 172.21.15.78:0/2782360442 >> 172.21.15.78:0/2782360442 conn(0x7f017c0ff710 msgr2=0x7f017c101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.598+0000 7f0181a73700 1 -- 172.21.15.78:0/2782360442 shutdown_connections 2023-07-14T18:56:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.598+0000 7f0181a73700 1 -- 172.21.15.78:0/2782360442 wait complete. 2023-07-14T18:56:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.598+0000 7f0181a73700 1 Processor -- start 2023-07-14T18:56:33.342 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.599+0000 7f0181a73700 1 -- start start 2023-07-14T18:56:33.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.599+0000 7f0181a73700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c18f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:33.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.599+0000 7f0181a73700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c18f730 con 0x7f017c104080 2023-07-14T18:56:33.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.599+0000 7f017b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c18f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:33.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.599+0000 7f017b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c18f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33438/0 (socket says 172.21.15.78:33438) 2023-07-14T18:56:33.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.599+0000 7f017b7fe700 1 -- 172.21.15.78:0/4141350315 learned_addr learned my addr 172.21.15.78:0/4141350315 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:33.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.599+0000 7f017b7fe700 1 -- 172.21.15.78:0/4141350315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c009550 con 0x7f017c104080 2023-07-14T18:56:33.343 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.599+0000 7f017b7fe700 1 --2- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c18f1f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f016c006af0 tx=0x7f016c011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:33.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.600+0000 7f0178ff9700 1 -- 172.21.15.78:0/4141350315 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f016c0037d0 con 0x7f017c104080 2023-07-14T18:56:33.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.600+0000 7f0178ff9700 1 -- 172.21.15.78:0/4141350315 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f016c01b440 con 0x7f017c104080 2023-07-14T18:56:33.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.600+0000 7f0178ff9700 1 -- 172.21.15.78:0/4141350315 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f016c0193d0 con 0x7f017c104080 2023-07-14T18:56:33.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.600+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017c18d840 con 0x7f017c104080 2023-07-14T18:56:33.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.600+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c18dc60 con 0x7f017c104080 2023-07-14T18:56:33.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.601+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017c061c70 con 0x7f017c104080 2023-07-14T18:56:33.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.601+0000 7f0178ff9700 1 -- 172.21.15.78:0/4141350315 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f016c019690 con 0x7f017c104080 2023-07-14T18:56:33.344 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.601+0000 7f0178ff9700 1 --2- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f01640387d0 0x7f016403ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:33.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.601+0000 7f0178ff9700 1 -- 172.21.15.78:0/4141350315 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f016c04c830 con 0x7f017c104080 2023-07-14T18:56:33.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.601+0000 7f017affd700 1 --2- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f01640387d0 0x7f016403ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:33.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.601+0000 7f017affd700 1 --2- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f01640387d0 0x7f016403ac80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0170006fd0 tx=0x7f0170006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:33.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.606+0000 7f0178ff9700 1 -- 172.21.15.78:0/4141350315 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f016c018db0 con 0x7f017c104080 2023-07-14T18:56:33.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.804+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f017c1017e0 con 0x7f01640387d0 2023-07-14T18:56:33.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.962+0000 7f0178ff9700 1 -- 172.21.15.78:0/4141350315 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f017c1017e0 con 0x7f01640387d0 2023-07-14T18:56:33.345 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.964+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f01640387d0 msgr2=0x7f016403ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:33.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.964+0000 7f0181a73700 1 --2- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f01640387d0 0x7f016403ac80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0170006fd0 tx=0x7f0170006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:33.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.964+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 msgr2=0x7f017c18f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:33.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.964+0000 7f0181a73700 1 --2- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c18f1f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f016c006af0 tx=0x7f016c011e80 comp rx=0 tx=0).stop 2023-07-14T18:56:33.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.964+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 shutdown_connections 2023-07-14T18:56:33.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.964+0000 7f0181a73700 1 --2- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f01640387d0 0x7f016403ac80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:33.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.964+0000 7f0181a73700 1 --2- 172.21.15.78:0/4141350315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f017c104080 0x7f017c18f1f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:33.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.965+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 >> 172.21.15.78:0/4141350315 conn(0x7f017c0ff710 msgr2=0x7f017c0798d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:33.346 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.965+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 shutdown_connections 2023-07-14T18:56:33.347 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:32.965+0000 7f0181a73700 1 -- 172.21.15.78:0/4141350315 wait complete. 2023-07-14T18:56:33.347 INFO:teuthology.orchestra.run.smithi078.stdout:Creating initial admin user... 2023-07-14T18:56:34.359 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:34 smithi078 ceph-mon[127425]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:34.360 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:34 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:34.885 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$tIUBrxh7n54BcjekMfuKSOObFNTk1Ixbfinu70H8lsjRXlXuUMe7.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1689360994, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-07-14T18:56:34.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.786+0000 7f1eaa1bf700 1 Processor -- start 2023-07-14T18:56:34.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.786+0000 7f1eaa1bf700 1 -- start start 2023-07-14T18:56:34.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1eaa1bf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea4079cd0 0x7f1ea410a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:34.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1eaa1bf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea4105b10 con 0x7f1ea4079cd0 2023-07-14T18:56:34.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1ea37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea4079cd0 0x7f1ea410a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:34.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1ea37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea4079cd0 0x7f1ea410a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33442/0 (socket says 172.21.15.78:33442) 2023-07-14T18:56:34.886 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1ea37fe700 1 -- 172.21.15.78:0/1753158334 learned_addr learned my addr 172.21.15.78:0/1753158334 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:34.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1ea37fe700 1 -- 172.21.15.78:0/1753158334 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea410a660 con 0x7f1ea4079cd0 2023-07-14T18:56:34.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1ea37fe700 1 --2- 172.21.15.78:0/1753158334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea4079cd0 0x7f1ea410a120 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009e80 tx=0x7f1e9c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c77baa9e7211ca6d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:34.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1ea27fc700 1 -- 172.21.15.78:0/1753158334 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e9c00e070 con 0x7f1ea4079cd0 2023-07-14T18:56:34.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1ea27fc700 1 -- 172.21.15.78:0/1753158334 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1e9c00b8c0 con 0x7f1ea4079cd0 2023-07-14T18:56:34.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.787+0000 7f1ea27fc700 1 -- 172.21.15.78:0/1753158334 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e9c004360 con 0x7f1ea4079cd0 2023-07-14T18:56:34.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.788+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/1753158334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea4079cd0 msgr2=0x7f1ea410a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:34.887 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.788+0000 7f1eaa1bf700 1 --2- 172.21.15.78:0/1753158334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea4079cd0 0x7f1ea410a120 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c009e80 tx=0x7f1e9c00b330 comp rx=0 tx=0).stop 2023-07-14T18:56:34.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.788+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/1753158334 shutdown_connections 2023-07-14T18:56:34.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.788+0000 7f1eaa1bf700 1 --2- 172.21.15.78:0/1753158334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea4079cd0 0x7f1ea410a120 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:34.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.788+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/1753158334 >> 172.21.15.78:0/1753158334 conn(0x7f1ea40ff710 msgr2=0x7f1ea4101b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:34.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.788+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/1753158334 shutdown_connections 2023-07-14T18:56:34.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.788+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/1753158334 wait complete. 2023-07-14T18:56:34.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.789+0000 7f1eaa1bf700 1 Processor -- start 2023-07-14T18:56:34.888 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.789+0000 7f1eaa1bf700 1 -- start start 2023-07-14T18:56:34.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.789+0000 7f1eaa1bf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea419af60 0x7f1ea419b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:34.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.789+0000 7f1eaa1bf700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea419b8b0 con 0x7f1ea419af60 2023-07-14T18:56:34.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.789+0000 7f1ea37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea419af60 0x7f1ea419b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:34.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.789+0000 7f1ea37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea419af60 0x7f1ea419b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33444/0 (socket says 172.21.15.78:33444) 2023-07-14T18:56:34.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.789+0000 7f1ea37fe700 1 -- 172.21.15.78:0/2764553937 learned_addr learned my addr 172.21.15.78:0/2764553937 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:34.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.790+0000 7f1ea37fe700 1 -- 172.21.15.78:0/2764553937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9c009b30 con 0x7f1ea419af60 2023-07-14T18:56:34.889 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.790+0000 7f1ea37fe700 1 --2- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea419af60 0x7f1ea419b370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0072a0 tx=0x7f1e9c003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:34.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.790+0000 7f1ea0ff9700 1 -- 172.21.15.78:0/2764553937 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e9c00e040 con 0x7f1ea419af60 2023-07-14T18:56:34.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.790+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea419bab0 con 0x7f1ea419af60 2023-07-14T18:56:34.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.790+0000 7f1ea0ff9700 1 -- 172.21.15.78:0/2764553937 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1e9c00bd70 con 0x7f1ea419af60 2023-07-14T18:56:34.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.790+0000 7f1ea0ff9700 1 -- 172.21.15.78:0/2764553937 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1e9c01c590 con 0x7f1ea419af60 2023-07-14T18:56:34.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.790+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea419e710 con 0x7f1ea419af60 2023-07-14T18:56:34.890 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.791+0000 7f1ea0ff9700 1 -- 172.21.15.78:0/2764553937 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7f1e9c01c850 con 0x7f1ea419af60 2023-07-14T18:56:34.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.791+0000 7f1ea0ff9700 1 --2- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e900387b0 0x7f1e9003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:34.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.791+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea4061c70 con 0x7f1ea419af60 2023-07-14T18:56:34.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.791+0000 7f1ea0ff9700 1 -- 172.21.15.78:0/2764553937 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1e9c04c920 con 0x7f1ea419af60 2023-07-14T18:56:34.891 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.791+0000 7f1ea2ffd700 1 --2- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e900387b0 0x7f1e9003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:34.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.791+0000 7f1ea2ffd700 1 --2- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e900387b0 0x7f1e9003ac60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c006fd0 tx=0x7f1e8c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:34.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:33.796+0000 7f1ea0ff9700 1 -- 172.21.15.78:0/2764553937 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1e9c024ba0 con 0x7f1ea419af60 2023-07-14T18:56:34.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.015+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f1ea4079cd0 con 0x7f1e900387b0 2023-07-14T18:56:34.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.242+0000 7f1ea0ff9700 1 -- 172.21.15.78:0/2764553937 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f1ea4079cd0 con 0x7f1e900387b0 2023-07-14T18:56:34.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.244+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e900387b0 msgr2=0x7f1e9003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:34.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.244+0000 7f1eaa1bf700 1 --2- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e900387b0 0x7f1e9003ac60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1e8c006fd0 tx=0x7f1e8c006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:34.892 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.244+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea419af60 msgr2=0x7f1ea419b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:34.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.244+0000 7f1eaa1bf700 1 --2- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea419af60 0x7f1ea419b370 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c0072a0 tx=0x7f1e9c003d00 comp rx=0 tx=0).stop 2023-07-14T18:56:34.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.244+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 shutdown_connections 2023-07-14T18:56:34.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.244+0000 7f1eaa1bf700 1 --2- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e900387b0 0x7f1e9003ac60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:34.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.245+0000 7f1eaa1bf700 1 --2- 172.21.15.78:0/2764553937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ea419af60 0x7f1ea419b370 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:34.893 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.245+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 >> 172.21.15.78:0/2764553937 conn(0x7f1ea40ff710 msgr2=0x7f1ea41003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:34.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.245+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 shutdown_connections 2023-07-14T18:56:34.894 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:34.245+0000 7f1eaa1bf700 1 -- 172.21.15.78:0/2764553937 wait complete. 2023-07-14T18:56:34.894 INFO:teuthology.orchestra.run.smithi078.stdout:Fetching dashboard port number... 2023-07-14T18:56:35.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:35 smithi078 ceph-mon[127425]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:35.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:36.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stdout 8443 2023-07-14T18:56:36.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.537+0000 7fa6d52d0700 1 Processor -- start 2023-07-14T18:56:36.083 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.537+0000 7fa6d52d0700 1 -- start start 2023-07-14T18:56:36.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.537+0000 7fa6d52d0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0105a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:36.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.537+0000 7fa6d52d0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d0105fa0 con 0x7fa6d0105650 2023-07-14T18:56:36.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.537+0000 7fa6ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0105a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:36.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.537+0000 7fa6ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0105a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33452/0 (socket says 172.21.15.78:33452) 2023-07-14T18:56:36.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.537+0000 7fa6ceffd700 1 -- 172.21.15.78:0/2141274165 learned_addr learned my addr 172.21.15.78:0/2141274165 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:36.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6ceffd700 1 -- 172.21.15.78:0/2141274165 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d01060e0 con 0x7fa6d0105650 2023-07-14T18:56:36.084 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6ceffd700 1 --2- 172.21.15.78:0/2141274165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0105a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40098a0 tx=0x7fa6c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba65525aba712bb1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:36.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6cdffb700 1 -- 172.21.15.78:0/2141274165 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6c4004160 con 0x7fa6d0105650 2023-07-14T18:56:36.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6cdffb700 1 -- 172.21.15.78:0/2141274165 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa6c40042c0 con 0x7fa6d0105650 2023-07-14T18:56:36.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6cdffb700 1 -- 172.21.15.78:0/2141274165 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6c40118e0 con 0x7fa6d0105650 2023-07-14T18:56:36.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2141274165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 msgr2=0x7fa6d0105a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:36.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6d52d0700 1 --2- 172.21.15.78:0/2141274165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0105a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40098a0 tx=0x7fa6c4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:36.085 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2141274165 shutdown_connections 2023-07-14T18:56:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6d52d0700 1 --2- 172.21.15.78:0/2141274165 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0105a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2141274165 >> 172.21.15.78:0/2141274165 conn(0x7fa6d00fee80 msgr2=0x7fa6d0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2141274165 shutdown_connections 2023-07-14T18:56:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.538+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2141274165 wait complete. 2023-07-14T18:56:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.539+0000 7fa6d52d0700 1 Processor -- start 2023-07-14T18:56:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.539+0000 7fa6d52d0700 1 -- start start 2023-07-14T18:56:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6d52d0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0198d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:36.086 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6d52d0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d0199270 con 0x7fa6d0105650 2023-07-14T18:56:36.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0198d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:36.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6ceffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0198d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33454/0 (socket says 172.21.15.78:33454) 2023-07-14T18:56:36.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6ceffd700 1 -- 172.21.15.78:0/2187525435 learned_addr learned my addr 172.21.15.78:0/2187525435 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:36.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6ceffd700 1 -- 172.21.15.78:0/2187525435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c4009550 con 0x7fa6d0105650 2023-07-14T18:56:36.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6ceffd700 1 --2- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0198d30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40118f0 tx=0x7fa6c40119d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:36.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6b7fff700 1 -- 172.21.15.78:0/2187525435 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6c40036a0 con 0x7fa6d0105650 2023-07-14T18:56:36.087 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6b7fff700 1 -- 172.21.15.78:0/2187525435 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa6c401b440 con 0x7fa6d0105650 2023-07-14T18:56:36.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d0199470 con 0x7fa6d0105650 2023-07-14T18:56:36.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6b7fff700 1 -- 172.21.15.78:0/2187525435 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa6c40193f0 con 0x7fa6d0105650 2023-07-14T18:56:36.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.540+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d0199890 con 0x7fa6d0105650 2023-07-14T18:56:36.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.541+0000 7fa6b7fff700 1 -- 172.21.15.78:0/2187525435 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 12) v1 ==== 45496+0+0 (secure 0 0 0) 0x7fa6c401b5b0 con 0x7fa6d0105650 2023-07-14T18:56:36.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.541+0000 7fa6b7fff700 1 --2- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa6b8038870 0x7fa6b803ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:36.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.541+0000 7fa6b7fff700 1 -- 172.21.15.78:0/2187525435 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa6c404c6c0 con 0x7fa6d0105650 2023-07-14T18:56:36.088 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.541+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d0061c70 con 0x7fa6d0105650 2023-07-14T18:56:36.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.541+0000 7fa6ce7fc700 1 --2- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa6b8038870 0x7fa6b803ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:36.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.542+0000 7fa6ce7fc700 1 --2- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa6b8038870 0x7fa6b803ad20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0006fd0 tx=0x7fa6c0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:36.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.546+0000 7fa6b7fff700 1 -- 172.21.15.78:0/2187525435 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa6c401f020 con 0x7fa6d0105650 2023-07-14T18:56:36.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.730+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fa6d0104f00 con 0x7fa6d0105650 2023-07-14T18:56:36.089 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.731+0000 7fa6b7fff700 1 -- 172.21.15.78:0/2187525435 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fa6c4003800 con 0x7fa6d0105650 2023-07-14T18:56:36.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa6b8038870 msgr2=0x7fa6b803ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:36.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 --2- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa6b8038870 0x7fa6b803ad20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0006fd0 tx=0x7fa6c0006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:36.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 msgr2=0x7fa6d0198d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:36.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 --2- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0198d30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa6c40118f0 tx=0x7fa6c40119d0 comp rx=0 tx=0).stop 2023-07-14T18:56:36.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 shutdown_connections 2023-07-14T18:56:36.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 --2- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa6b8038870 0x7fa6b803ad20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:36.090 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 --2- 172.21.15.78:0/2187525435 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6d0105650 0x7fa6d0198d30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:36.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 >> 172.21.15.78:0/2187525435 conn(0x7fa6d00fee80 msgr2=0x7fa6d0103b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:36.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 shutdown_connections 2023-07-14T18:56:36.091 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:35.733+0000 7fa6d52d0700 1 -- 172.21.15.78:0/2187525435 wait complete. 2023-07-14T18:56:36.092 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-07-14T18:56:36.093 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout disabled 2023-07-14T18:56:36.098 INFO:teuthology.orchestra.run.smithi078.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-07-14T18:56:36.098 INFO:teuthology.orchestra.run.smithi078.stdout:systemctl: stdout inactive 2023-07-14T18:56:36.098 INFO:teuthology.orchestra.run.smithi078.stdout:firewalld.service is not enabled 2023-07-14T18:56:36.098 INFO:teuthology.orchestra.run.smithi078.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-07-14T18:56:36.099 INFO:teuthology.orchestra.run.smithi078.stdout:Ceph Dashboard is now available at: 2023-07-14T18:56:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: URL: https://smithi078.front.sepia.ceph.com:8443/ 2023-07-14T18:56:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: User: admin 2023-07-14T18:56:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: Password: e461hm1xtn 2023-07-14T18:56:36.100 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:36.101 INFO:teuthology.orchestra.run.smithi078.stdout:Saving cluster configuration to /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config directory 2023-07-14T18:56:36.101 INFO:teuthology.orchestra.run.smithi078.stdout:Enabling autotune for osd_memory_target 2023-07-14T18:56:36.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:36 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2187525435' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-07-14T18:56:37.146 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64dc7d4700 1 Processor -- start 2023-07-14T18:56:37.147 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64dc7d4700 1 -- start start 2023-07-14T18:56:37.147 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64dc7d4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d41080a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:37.147 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64dc7d4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4079d10 con 0x7f64d4105cc0 2023-07-14T18:56:37.147 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64da570700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d41080a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:37.147 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64da570700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d41080a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33458/0 (socket says 172.21.15.78:33458) 2023-07-14T18:56:37.147 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64da570700 1 -- 172.21.15.78:0/1590229968 learned_addr learned my addr 172.21.15.78:0/1590229968 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:37.147 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64da570700 1 -- 172.21.15.78:0/1590229968 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d4079e50 con 0x7f64d4105cc0 2023-07-14T18:56:37.148 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.543+0000 7f64da570700 1 --2- 172.21.15.78:0/1590229968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d41080a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f64c40098a0 tx=0x7f64c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebff5f51659c0645 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:37.148 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64d956e700 1 -- 172.21.15.78:0/1590229968 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64c4004160 con 0x7f64d4105cc0 2023-07-14T18:56:37.148 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64d956e700 1 -- 172.21.15.78:0/1590229968 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f64c40042c0 con 0x7f64d4105cc0 2023-07-14T18:56:37.148 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64d956e700 1 -- 172.21.15.78:0/1590229968 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64c40118e0 con 0x7f64d4105cc0 2023-07-14T18:56:37.148 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64dc7d4700 1 -- 172.21.15.78:0/1590229968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 msgr2=0x7f64d41080a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:37.148 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64dc7d4700 1 --2- 172.21.15.78:0/1590229968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d41080a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f64c40098a0 tx=0x7f64c4009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:37.148 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64dc7d4700 1 -- 172.21.15.78:0/1590229968 shutdown_connections 2023-07-14T18:56:37.149 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64dc7d4700 1 --2- 172.21.15.78:0/1590229968 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d41080a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:37.149 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64dc7d4700 1 -- 172.21.15.78:0/1590229968 >> 172.21.15.78:0/1590229968 conn(0x7f64d40ff6e0 msgr2=0x7f64d4101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:37.149 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64dc7d4700 1 -- 172.21.15.78:0/1590229968 shutdown_connections 2023-07-14T18:56:37.149 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.544+0000 7f64dc7d4700 1 -- 172.21.15.78:0/1590229968 wait complete. 2023-07-14T18:56:37.149 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64dc7d4700 1 Processor -- start 2023-07-14T18:56:37.149 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64dc7d4700 1 -- start start 2023-07-14T18:56:37.150 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64dc7d4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d4105620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:37.150 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64dc7d4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d4105b60 con 0x7f64d4105cc0 2023-07-14T18:56:37.150 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64da570700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d4105620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:37.150 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64da570700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d4105620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33460/0 (socket says 172.21.15.78:33460) 2023-07-14T18:56:37.150 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64da570700 1 -- 172.21.15.78:0/97925521 learned_addr learned my addr 172.21.15.78:0/97925521 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:37.150 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64da570700 1 -- 172.21.15.78:0/97925521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c4009550 con 0x7f64d4105cc0 2023-07-14T18:56:37.150 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.545+0000 7f64da570700 1 --2- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d4105620 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64c4000c00 tx=0x7f64c40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:37.151 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.546+0000 7f64cb7fe700 1 -- 172.21.15.78:0/97925521 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64c40036a0 con 0x7f64d4105cc0 2023-07-14T18:56:37.151 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.546+0000 7f64cb7fe700 1 -- 172.21.15.78:0/97925521 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f64c401b440 con 0x7f64d4105cc0 2023-07-14T18:56:37.151 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.546+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d4103c70 con 0x7f64d4105cc0 2023-07-14T18:56:37.151 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.546+0000 7f64cb7fe700 1 -- 172.21.15.78:0/97925521 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64c401a3d0 con 0x7f64d4105cc0 2023-07-14T18:56:37.151 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.546+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d41040f0 con 0x7f64d4105cc0 2023-07-14T18:56:37.151 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.546+0000 7f64cb7fe700 1 -- 172.21.15.78:0/97925521 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f64c401b5b0 con 0x7f64d4105cc0 2023-07-14T18:56:37.152 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.547+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d41091d0 con 0x7f64d4105cc0 2023-07-14T18:56:37.152 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.547+0000 7f64cb7fe700 1 --2- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f64c00388c0 0x7f64c003ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:37.152 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.547+0000 7f64cb7fe700 1 -- 172.21.15.78:0/97925521 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f64c404c770 con 0x7f64d4105cc0 2023-07-14T18:56:37.152 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.547+0000 7f64d9d6f700 1 --2- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f64c00388c0 0x7f64c003ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:37.152 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.547+0000 7f64d9d6f700 1 --2- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f64c00388c0 0x7f64c003ad70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f64d0006fd0 tx=0x7f64d0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:37.152 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.551+0000 7f64cb7fe700 1 -- 172.21.15.78:0/97925521 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f64c4023590 con 0x7f64d4105cc0 2023-07-14T18:56:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.735+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f64d41093f0 con 0x7f64d4105cc0 2023-07-14T18:56:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.735+0000 7f64cb7fe700 1 -- 172.21.15.78:0/97925521 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f64c401f030 con 0x7f64d4105cc0 2023-07-14T18:56:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.737+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f64c00388c0 msgr2=0x7f64c003ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.737+0000 7f64dc7d4700 1 --2- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f64c00388c0 0x7f64c003ad70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f64d0006fd0 tx=0x7f64d0006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.737+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 msgr2=0x7f64d4105620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.737+0000 7f64dc7d4700 1 --2- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d4105620 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f64c4000c00 tx=0x7f64c40119a0 comp rx=0 tx=0).stop 2023-07-14T18:56:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.738+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 shutdown_connections 2023-07-14T18:56:37.153 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.738+0000 7f64dc7d4700 1 --2- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f64c00388c0 0x7f64c003ad70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:37.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.738+0000 7f64dc7d4700 1 --2- 172.21.15.78:0/97925521 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f64d4105cc0 0x7f64d4105620 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:37.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.738+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 >> 172.21.15.78:0/97925521 conn(0x7f64d40ff6e0 msgr2=0x7f64d41003b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:37.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.738+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 shutdown_connections 2023-07-14T18:56:37.154 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:36.738+0000 7f64dc7d4700 1 -- 172.21.15.78:0/97925521 wait complete. 2023-07-14T18:56:37.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:37 smithi078 ceph-mon[127425]: mgrmap e13: x(active, since 6s) 2023-07-14T18:56:38.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.628+0000 7fa11e8a3700 1 Processor -- start 2023-07-14T18:56:38.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.628+0000 7fa11e8a3700 1 -- start start 2023-07-14T18:56:38.290 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.628+0000 7fa11e8a3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa118079cd0 0x7fa11810a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:38.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.628+0000 7fa11e8a3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa118105b00 con 0x7fa118079cd0 2023-07-14T18:56:38.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.629+0000 7fa117fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa118079cd0 0x7fa11810a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:38.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.629+0000 7fa117fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa118079cd0 0x7fa11810a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33464/0 (socket says 172.21.15.78:33464) 2023-07-14T18:56:38.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.629+0000 7fa117fff700 1 -- 172.21.15.78:0/12490940 learned_addr learned my addr 172.21.15.78:0/12490940 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:38.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.629+0000 7fa117fff700 1 -- 172.21.15.78:0/12490940 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11810a650 con 0x7fa118079cd0 2023-07-14T18:56:38.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.629+0000 7fa117fff700 1 --2- 172.21.15.78:0/12490940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa118079cd0 0x7fa11810a110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0098a0 tx=0x7fa10c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16e6025051ac35e9 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:38.291 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.629+0000 7fa116ffd700 1 -- 172.21.15.78:0/12490940 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa10c004160 con 0x7fa118079cd0 2023-07-14T18:56:38.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.629+0000 7fa116ffd700 1 -- 172.21.15.78:0/12490940 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa10c0042c0 con 0x7fa118079cd0 2023-07-14T18:56:38.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.629+0000 7fa116ffd700 1 -- 172.21.15.78:0/12490940 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa10c0118e0 con 0x7fa118079cd0 2023-07-14T18:56:38.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 -- 172.21.15.78:0/12490940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa118079cd0 msgr2=0x7fa11810a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:38.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 --2- 172.21.15.78:0/12490940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa118079cd0 0x7fa11810a110 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0098a0 tx=0x7fa10c009bb0 comp rx=0 tx=0).stop 2023-07-14T18:56:38.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 -- 172.21.15.78:0/12490940 shutdown_connections 2023-07-14T18:56:38.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 --2- 172.21.15.78:0/12490940 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa118079cd0 0x7fa11810a110 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:38.292 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 -- 172.21.15.78:0/12490940 >> 172.21.15.78:0/12490940 conn(0x7fa1180ff6e0 msgr2=0x7fa118101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 -- 172.21.15.78:0/12490940 shutdown_connections 2023-07-14T18:56:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 -- 172.21.15.78:0/12490940 wait complete. 2023-07-14T18:56:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 Processor -- start 2023-07-14T18:56:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.630+0000 7fa11e8a3700 1 -- start start 2023-07-14T18:56:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa11e8a3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa11819af20 0x7fa11819b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa11e8a3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa11819b870 con 0x7fa11819af20 2023-07-14T18:56:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa117fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa11819af20 0x7fa11819b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:38.293 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa117fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa11819af20 0x7fa11819b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33466/0 (socket says 172.21.15.78:33466) 2023-07-14T18:56:38.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa117fff700 1 -- 172.21.15.78:0/3287682984 learned_addr learned my addr 172.21.15.78:0/3287682984 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:38.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa117fff700 1 -- 172.21.15.78:0/3287682984 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10c009550 con 0x7fa11819af20 2023-07-14T18:56:38.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa117fff700 1 --2- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa11819af20 0x7fa11819b330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa10c000c00 tx=0x7fa10c0119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:38.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa1157fa700 1 -- 172.21.15.78:0/3287682984 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa10c0036a0 con 0x7fa11819af20 2023-07-14T18:56:38.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa1157fa700 1 -- 172.21.15.78:0/3287682984 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa10c01b440 con 0x7fa11819af20 2023-07-14T18:56:38.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa11819ba70 con 0x7fa11819af20 2023-07-14T18:56:38.294 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa1157fa700 1 -- 172.21.15.78:0/3287682984 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa10c01a3d0 con 0x7fa11819af20 2023-07-14T18:56:38.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.631+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa11819e6d0 con 0x7fa11819af20 2023-07-14T18:56:38.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.632+0000 7fa1157fa700 1 -- 172.21.15.78:0/3287682984 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fa10c01a640 con 0x7fa11819af20 2023-07-14T18:56:38.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.632+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa118079cd0 con 0x7fa11819af20 2023-07-14T18:56:38.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.632+0000 7fa1157fa700 1 --2- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa100038910 0x7fa10003adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:38.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.632+0000 7fa1157fa700 1 -- 172.21.15.78:0/3287682984 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fa10c04c740 con 0x7fa11819af20 2023-07-14T18:56:38.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.633+0000 7fa1177fe700 1 --2- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa100038910 0x7fa10003adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:38.295 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.633+0000 7fa1177fe700 1 --2- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa100038910 0x7fa10003adc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa108006fd0 tx=0x7fa108006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:38.296 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.637+0000 7fa1157fa700 1 -- 172.21.15.78:0/3287682984 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa10c023770 con 0x7fa11819af20 2023-07-14T18:56:38.296 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.925+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fa118079ef0 con 0x7fa11819af20 2023-07-14T18:56:38.296 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.928+0000 7fa1157fa700 1 -- 172.21.15.78:0/3287682984 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fa10c01f030 con 0x7fa11819af20 2023-07-14T18:56:38.296 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.930+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa100038910 msgr2=0x7fa10003adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:38.296 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.930+0000 7fa11e8a3700 1 --2- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa100038910 0x7fa10003adc0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa108006fd0 tx=0x7fa108006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:38.296 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.930+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa11819af20 msgr2=0x7fa11819b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:38.296 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.930+0000 7fa11e8a3700 1 --2- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa11819af20 0x7fa11819b330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fa10c000c00 tx=0x7fa10c0119a0 comp rx=0 tx=0).stop 2023-07-14T18:56:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.930+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 shutdown_connections 2023-07-14T18:56:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.930+0000 7fa11e8a3700 1 --2- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa100038910 0x7fa10003adc0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.930+0000 7fa11e8a3700 1 --2- 172.21.15.78:0/3287682984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa11819af20 0x7fa11819b330 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.930+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 >> 172.21.15.78:0/3287682984 conn(0x7fa1180ff6e0 msgr2=0x7fa1181003a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.931+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 shutdown_connections 2023-07-14T18:56:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr 2023-07-14T18:56:37.931+0000 7fa11e8a3700 1 -- 172.21.15.78:0/3287682984 wait complete. 2023-07-14T18:56:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-07-14T18:56:38.297 INFO:teuthology.orchestra.run.smithi078.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-07-14T18:56:38.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:38.298 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-07-14T18:56:38.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:38.298 INFO:teuthology.orchestra.run.smithi078.stdout:Or, if you are only running a single cluster on this host: 2023-07-14T18:56:38.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:38.298 INFO:teuthology.orchestra.run.smithi078.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-07-14T18:56:38.298 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:38.298 INFO:teuthology.orchestra.run.smithi078.stdout:Please consider enabling telemetry to help improve Ceph: 2023-07-14T18:56:38.299 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:38.299 INFO:teuthology.orchestra.run.smithi078.stdout: ceph telemetry on 2023-07-14T18:56:38.299 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:38.299 INFO:teuthology.orchestra.run.smithi078.stdout:For more information see: 2023-07-14T18:56:38.299 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:38.299 INFO:teuthology.orchestra.run.smithi078.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-07-14T18:56:38.299 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:38.299 INFO:teuthology.orchestra.run.smithi078.stdout:Bootstrap complete. 2023-07-14T18:56:38.324 INFO:tasks.cephadm:Fetching config... 2023-07-14T18:56:38.325 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:56:38.325 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-07-14T18:56:38.341 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-07-14T18:56:38.341 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:56:38.342 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-07-14T18:56:38.400 INFO:tasks.cephadm:Fetching mon keyring... 2023-07-14T18:56:38.400 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:56:38.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/keyring of=/dev/stdout 2023-07-14T18:56:38.468 INFO:tasks.cephadm:Fetching pub ssh key... 2023-07-14T18:56:38.468 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:56:38.468 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-07-14T18:56:38.524 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-07-14T18:56:38.525 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr3JZMYQ+W65+lp46KnvX8Bg7p0lyisZP7Ia/muor73+TjY8g5Muc6Dasm2Bd6O8pOWlP42+hDUw7bHs+lVzD+rYtjyEYicc49nbXIK807sKbsByWNOjRkwpL+NS86147vgON/TA+pXMW9c6TTbHW3hD/LvON7iY1zsK8Qom5kgIkSYvQF+9rt0X17vC9wDtFGgDd1fyKOMvuG2Tv7GEB1c7P3AFW32ZOuhAfaLBaEaWrzQuaJ8sNMCNgyy6HTZi7XsLn6NCEbR46xVXx8fYr7HWOzXDg4uWYsm2txloNXm5p8wumuson4cfBWnCdP4nNdil4plhVabHvH5CZAbs7nJSOpCQXvQDdCYR9Zgmo57AAIf1Z16QcaSzSpS9jE3Ty4VHlBSzf43z7FntKoS3Zr/b8LvCnk0DGipRfW4IHdT70q+WfAMOcEpEq5kfwW49Ogg5VZsypU/qDlHo2hnDUfxUJQ5imBtNyWurysK05SQudIPEMusbUuKTqBjm22gU= ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:56:38.600 INFO:teuthology.orchestra.run.smithi078.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr3JZMYQ+W65+lp46KnvX8Bg7p0lyisZP7Ia/muor73+TjY8g5Muc6Dasm2Bd6O8pOWlP42+hDUw7bHs+lVzD+rYtjyEYicc49nbXIK807sKbsByWNOjRkwpL+NS86147vgON/TA+pXMW9c6TTbHW3hD/LvON7iY1zsK8Qom5kgIkSYvQF+9rt0X17vC9wDtFGgDd1fyKOMvuG2Tv7GEB1c7P3AFW32ZOuhAfaLBaEaWrzQuaJ8sNMCNgyy6HTZi7XsLn6NCEbR46xVXx8fYr7HWOzXDg4uWYsm2txloNXm5p8wumuson4cfBWnCdP4nNdil4plhVabHvH5CZAbs7nJSOpCQXvQDdCYR9Zgmo57AAIf1Z16QcaSzSpS9jE3Ty4VHlBSzf43z7FntKoS3Zr/b8LvCnk0DGipRfW4IHdT70q+WfAMOcEpEq5kfwW49Ogg5VZsypU/qDlHo2hnDUfxUJQ5imBtNyWurysK05SQudIPEMusbUuKTqBjm22gU= ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:56:38.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr3JZMYQ+W65+lp46KnvX8Bg7p0lyisZP7Ia/muor73+TjY8g5Muc6Dasm2Bd6O8pOWlP42+hDUw7bHs+lVzD+rYtjyEYicc49nbXIK807sKbsByWNOjRkwpL+NS86147vgON/TA+pXMW9c6TTbHW3hD/LvON7iY1zsK8Qom5kgIkSYvQF+9rt0X17vC9wDtFGgDd1fyKOMvuG2Tv7GEB1c7P3AFW32ZOuhAfaLBaEaWrzQuaJ8sNMCNgyy6HTZi7XsLn6NCEbR46xVXx8fYr7HWOzXDg4uWYsm2txloNXm5p8wumuson4cfBWnCdP4nNdil4plhVabHvH5CZAbs7nJSOpCQXvQDdCYR9Zgmo57AAIf1Z16QcaSzSpS9jE3Ty4VHlBSzf43z7FntKoS3Zr/b8LvCnk0DGipRfW4IHdT70q+WfAMOcEpEq5kfwW49Ogg5VZsypU/qDlHo2hnDUfxUJQ5imBtNyWurysK05SQudIPEMusbUuKTqBjm22gU= ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:56:38.656 INFO:teuthology.orchestra.run.smithi088.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr3JZMYQ+W65+lp46KnvX8Bg7p0lyisZP7Ia/muor73+TjY8g5Muc6Dasm2Bd6O8pOWlP42+hDUw7bHs+lVzD+rYtjyEYicc49nbXIK807sKbsByWNOjRkwpL+NS86147vgON/TA+pXMW9c6TTbHW3hD/LvON7iY1zsK8Qom5kgIkSYvQF+9rt0X17vC9wDtFGgDd1fyKOMvuG2Tv7GEB1c7P3AFW32ZOuhAfaLBaEaWrzQuaJ8sNMCNgyy6HTZi7XsLn6NCEbR46xVXx8fYr7HWOzXDg4uWYsm2txloNXm5p8wumuson4cfBWnCdP4nNdil4plhVabHvH5CZAbs7nJSOpCQXvQDdCYR9Zgmo57AAIf1Z16QcaSzSpS9jE3Ty4VHlBSzf43z7FntKoS3Zr/b8LvCnk0DGipRfW4IHdT70q+WfAMOcEpEq5kfwW49Ogg5VZsypU/qDlHo2hnDUfxUJQ5imBtNyWurysK05SQudIPEMusbUuKTqBjm22gU= ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:56:38.669 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr3JZMYQ+W65+lp46KnvX8Bg7p0lyisZP7Ia/muor73+TjY8g5Muc6Dasm2Bd6O8pOWlP42+hDUw7bHs+lVzD+rYtjyEYicc49nbXIK807sKbsByWNOjRkwpL+NS86147vgON/TA+pXMW9c6TTbHW3hD/LvON7iY1zsK8Qom5kgIkSYvQF+9rt0X17vC9wDtFGgDd1fyKOMvuG2Tv7GEB1c7P3AFW32ZOuhAfaLBaEaWrzQuaJ8sNMCNgyy6HTZi7XsLn6NCEbR46xVXx8fYr7HWOzXDg4uWYsm2txloNXm5p8wumuson4cfBWnCdP4nNdil4plhVabHvH5CZAbs7nJSOpCQXvQDdCYR9Zgmo57AAIf1Z16QcaSzSpS9jE3Ty4VHlBSzf43z7FntKoS3Zr/b8LvCnk0DGipRfW4IHdT70q+WfAMOcEpEq5kfwW49Ogg5VZsypU/qDlHo2hnDUfxUJQ5imBtNyWurysK05SQudIPEMusbUuKTqBjm22gU= ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-07-14T18:56:38.711 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDFr3JZMYQ+W65+lp46KnvX8Bg7p0lyisZP7Ia/muor73+TjY8g5Muc6Dasm2Bd6O8pOWlP42+hDUw7bHs+lVzD+rYtjyEYicc49nbXIK807sKbsByWNOjRkwpL+NS86147vgON/TA+pXMW9c6TTbHW3hD/LvON7iY1zsK8Qom5kgIkSYvQF+9rt0X17vC9wDtFGgDd1fyKOMvuG2Tv7GEB1c7P3AFW32ZOuhAfaLBaEaWrzQuaJ8sNMCNgyy6HTZi7XsLn6NCEbR46xVXx8fYr7HWOzXDg4uWYsm2txloNXm5p8wumuson4cfBWnCdP4nNdil4plhVabHvH5CZAbs7nJSOpCQXvQDdCYR9Zgmo57AAIf1Z16QcaSzSpS9jE3Ty4VHlBSzf43z7FntKoS3Zr/b8LvCnk0DGipRfW4IHdT70q+WfAMOcEpEq5kfwW49Ogg5VZsypU/qDlHo2hnDUfxUJQ5imBtNyWurysK05SQudIPEMusbUuKTqBjm22gU= ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:56:38.724 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-07-14T18:56:39.255 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:38 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3287682984' entity='client.admin' 2023-07-14T18:56:39.282 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:56:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.022+0000 7fb96e443700 1 -- 172.21.15.78:0/1756944969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 msgr2=0x7fb9680fea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.022+0000 7fb96e443700 1 --2- 172.21.15.78:0/1756944969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 0x7fb9680fea20 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb958009910 tx=0x7fb958009c20 comp rx=0 tx=0).stop 2023-07-14T18:56:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.022+0000 7fb96e443700 1 -- 172.21.15.78:0/1756944969 shutdown_connections 2023-07-14T18:56:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.022+0000 7fb96e443700 1 --2- 172.21.15.78:0/1756944969 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 0x7fb9680fea20 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.022+0000 7fb96e443700 1 -- 172.21.15.78:0/1756944969 >> 172.21.15.78:0/1756944969 conn(0x7fb9680f9ba0 msgr2=0x7fb9680fbff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.022+0000 7fb96e443700 1 -- 172.21.15.78:0/1756944969 shutdown_connections 2023-07-14T18:56:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.022+0000 7fb96e443700 1 -- 172.21.15.78:0/1756944969 wait complete. 2023-07-14T18:56:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.023+0000 7fb96e443700 1 Processor -- start 2023-07-14T18:56:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.023+0000 7fb96e443700 1 -- start start 2023-07-14T18:56:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.023+0000 7fb96e443700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 0x7fb9681947d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.023+0000 7fb96e443700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb968194d10 con 0x7fb9680fe610 2023-07-14T18:56:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.023+0000 7fb967fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 0x7fb9681947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.023+0000 7fb967fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 0x7fb9681947d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33472/0 (socket says 172.21.15.78:33472) 2023-07-14T18:56:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.023+0000 7fb967fff700 1 -- 172.21.15.78:0/3965448254 learned_addr learned my addr 172.21.15.78:0/3965448254 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.023+0000 7fb967fff700 1 -- 172.21.15.78:0/3965448254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9580095f0 con 0x7fb9680fe610 2023-07-14T18:56:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.024+0000 7fb967fff700 1 --2- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 0x7fb9681947d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb958006010 tx=0x7fb958005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.024+0000 7fb9657fa700 1 -- 172.21.15.78:0/3965448254 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb95800ba90 con 0x7fb9680fe610 2023-07-14T18:56:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.024+0000 7fb9657fa700 1 -- 172.21.15.78:0/3965448254 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb95800bbf0 con 0x7fb9680fe610 2023-07-14T18:56:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.024+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb968194f10 con 0x7fb9680fe610 2023-07-14T18:56:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.024+0000 7fb9657fa700 1 -- 172.21.15.78:0/3965448254 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb958017440 con 0x7fb9680fe610 2023-07-14T18:56:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.024+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9681953b0 con 0x7fb9680fe610 2023-07-14T18:56:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.025+0000 7fb9657fa700 1 -- 172.21.15.78:0/3965448254 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fb958017670 con 0x7fb9680fe610 2023-07-14T18:56:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.025+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96818e5e0 con 0x7fb9680fe610 2023-07-14T18:56:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.025+0000 7fb9657fa700 1 --2- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb9500387b0 0x7fb95003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.025+0000 7fb9657fa700 1 -- 172.21.15.78:0/3965448254 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb95804cb30 con 0x7fb9680fe610 2023-07-14T18:56:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.025+0000 7fb9677fe700 1 --2- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb9500387b0 0x7fb95003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.025+0000 7fb9677fe700 1 --2- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb9500387b0 0x7fb95003ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb95c006fd0 tx=0x7fb95c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.030+0000 7fb9657fa700 1 -- 172.21.15.78:0/3965448254 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb958016990 con 0x7fb9680fe610 2023-07-14T18:56:41.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.220+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fb96802c8e0 con 0x7fb9680fe610 2023-07-14T18:56:41.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.223+0000 7fb9657fa700 1 -- 172.21.15.78:0/3965448254 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb958027020 con 0x7fb9680fe610 2023-07-14T18:56:41.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb9500387b0 msgr2=0x7fb95003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 --2- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb9500387b0 0x7fb95003ac60 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb95c006fd0 tx=0x7fb95c006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 msgr2=0x7fb9681947d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 --2- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 0x7fb9681947d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb958006010 tx=0x7fb958005810 comp rx=0 tx=0).stop 2023-07-14T18:56:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 shutdown_connections 2023-07-14T18:56:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 --2- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb9500387b0 0x7fb95003ac60 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 --2- 172.21.15.78:0/3965448254 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb9680fe610 0x7fb9681947d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:41.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 >> 172.21.15.78:0/3965448254 conn(0x7fb9680f9ba0 msgr2=0x7fb9680fa870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:41.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 shutdown_connections 2023-07-14T18:56:41.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:41.225+0000 7fb96e443700 1 -- 172.21.15.78:0/3965448254 wait complete. 2023-07-14T18:56:41.671 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-07-14T18:56:41.672 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-07-14T18:56:42.220 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:56:42.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:42 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3965448254' entity='client.admin' 2023-07-14T18:56:42.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:42.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:42.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi078", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:56:42.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:42.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:42.483 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:43.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.582+0000 7fc50ccef700 1 -- 172.21.15.78:0/4212011984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 msgr2=0x7fc5080fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:43.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.582+0000 7fc50ccef700 1 --2- 172.21.15.78:0/4212011984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 0x7fc5080fea30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8009910 tx=0x7fc4f8009c20 comp rx=0 tx=0).stop 2023-07-14T18:56:43.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.582+0000 7fc50ccef700 1 -- 172.21.15.78:0/4212011984 shutdown_connections 2023-07-14T18:56:43.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.582+0000 7fc50ccef700 1 --2- 172.21.15.78:0/4212011984 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 0x7fc5080fea30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:43.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.582+0000 7fc50ccef700 1 -- 172.21.15.78:0/4212011984 >> 172.21.15.78:0/4212011984 conn(0x7fc5080f9bb0 msgr2=0x7fc5080fc000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:43.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.582+0000 7fc50ccef700 1 -- 172.21.15.78:0/4212011984 shutdown_connections 2023-07-14T18:56:43.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.582+0000 7fc50ccef700 1 -- 172.21.15.78:0/4212011984 wait complete. 2023-07-14T18:56:43.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50ccef700 1 Processor -- start 2023-07-14T18:56:43.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50ccef700 1 -- start start 2023-07-14T18:56:43.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50ccef700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 0x7fc5081936a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:43.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50ccef700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc508193be0 con 0x7fc5080fe620 2023-07-14T18:56:43.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 0x7fc5081936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:43.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 0x7fc5081936a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33478/0 (socket says 172.21.15.78:33478) 2023-07-14T18:56:43.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50659c700 1 -- 172.21.15.78:0/2035820074 learned_addr learned my addr 172.21.15.78:0/2035820074 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:43.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50659c700 1 -- 172.21.15.78:0/2035820074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f80095f0 con 0x7fc5080fe620 2023-07-14T18:56:43.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50659c700 1 --2- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 0x7fc5081936a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8006010 tx=0x7fc4f8005810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:43.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc4f77fe700 1 -- 172.21.15.78:0/2035820074 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4f800ba90 con 0x7fc5080fe620 2023-07-14T18:56:43.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc4f77fe700 1 -- 172.21.15.78:0/2035820074 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc4f800bbf0 con 0x7fc5080fe620 2023-07-14T18:56:43.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc4f77fe700 1 -- 172.21.15.78:0/2035820074 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc4f800f460 con 0x7fc5080fe620 2023-07-14T18:56:43.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.583+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc508193de0 con 0x7fc5080fe620 2023-07-14T18:56:43.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.584+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc508194280 con 0x7fc5080fe620 2023-07-14T18:56:43.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.584+0000 7fc4f77fe700 1 -- 172.21.15.78:0/2035820074 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fc4f800f6e0 con 0x7fc5080fe620 2023-07-14T18:56:43.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.584+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc508104210 con 0x7fc5080fe620 2023-07-14T18:56:43.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.584+0000 7fc4f77fe700 1 --2- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc4f00387b0 0x7fc4f003ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:43.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.585+0000 7fc4f77fe700 1 -- 172.21.15.78:0/2035820074 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fc4f804ff00 con 0x7fc5080fe620 2023-07-14T18:56:43.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.585+0000 7fc505d9b700 1 --2- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc4f00387b0 0x7fc4f003ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:43.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.585+0000 7fc505d9b700 1 --2- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc4f00387b0 0x7fc4f003ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc006fd0 tx=0x7fc4fc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:43.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.589+0000 7fc4f77fe700 1 -- 172.21.15.78:0/2035820074 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc4f8017440 con 0x7fc5080fe620 2023-07-14T18:56:43.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.777+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fc5080680d0 con 0x7fc4f00387b0 2023-07-14T18:56:43.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.780+0000 7fc4f77fe700 1 -- 172.21.15.78:0/2035820074 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc5080680d0 con 0x7fc4f00387b0 2023-07-14T18:56:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc4f00387b0 msgr2=0x7fc4f003ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 --2- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc4f00387b0 0x7fc4f003ac60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc4fc006fd0 tx=0x7fc4fc006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 msgr2=0x7fc5081936a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 --2- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 0x7fc5081936a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc4f8006010 tx=0x7fc4f8005810 comp rx=0 tx=0).stop 2023-07-14T18:56:43.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 shutdown_connections 2023-07-14T18:56:43.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 --2- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc4f00387b0 0x7fc4f003ac60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:43.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 --2- 172.21.15.78:0/2035820074 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5080fe620 0x7fc5081936a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:43.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 >> 172.21.15.78:0/2035820074 conn(0x7fc5080f9bb0 msgr2=0x7fc5080fa940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:43.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.782+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 shutdown_connections 2023-07-14T18:56:43.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:43.783+0000 7fc50ccef700 1 -- 172.21.15.78:0/2035820074 wait complete. 2023-07-14T18:56:44.277 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi088 2023-07-14T18:56:44.277 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:56:44.277 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.conf 2023-07-14T18:56:44.298 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:56:44.298 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:56:44.357 INFO:tasks.cephadm:Adding host smithi088 to orchestrator... 2023-07-14T18:56:44.358 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch host add smithi088 2023-07-14T18:56:44.918 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:56:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:56:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:56:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: Updating smithi078:/etc/ceph/ceph.conf 2023-07-14T18:56:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: Updating smithi078:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:56:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: Updating smithi078:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:56:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:46.033 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:45 smithi078 ceph-mon[127425]: Updating smithi078:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:56:46.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.250+0000 7fdec62c8700 1 -- 172.21.15.78:0/1950653072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 msgr2=0x7fdec00ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:46.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.250+0000 7fdec62c8700 1 --2- 172.21.15.78:0/1950653072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 0x7fdec00ffa60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0009910 tx=0x7fdeb0009c20 comp rx=0 tx=0).stop 2023-07-14T18:56:46.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.250+0000 7fdec62c8700 1 -- 172.21.15.78:0/1950653072 shutdown_connections 2023-07-14T18:56:46.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.250+0000 7fdec62c8700 1 --2- 172.21.15.78:0/1950653072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 0x7fdec00ffa60 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:46.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.250+0000 7fdec62c8700 1 -- 172.21.15.78:0/1950653072 >> 172.21.15.78:0/1950653072 conn(0x7fdec0075040 msgr2=0x7fdec0075440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:46.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.250+0000 7fdec62c8700 1 -- 172.21.15.78:0/1950653072 shutdown_connections 2023-07-14T18:56:46.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.250+0000 7fdec62c8700 1 -- 172.21.15.78:0/1950653072 wait complete. 2023-07-14T18:56:46.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.250+0000 7fdec62c8700 1 Processor -- start 2023-07-14T18:56:46.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.251+0000 7fdec62c8700 1 -- start start 2023-07-14T18:56:46.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.251+0000 7fdec62c8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 0x7fdec01959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:46.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.251+0000 7fdec62c8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec0195f20 con 0x7fdec00ff690 2023-07-14T18:56:46.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.251+0000 7fdebffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 0x7fdec01959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:46.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.251+0000 7fdebffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 0x7fdec01959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33484/0 (socket says 172.21.15.78:33484) 2023-07-14T18:56:46.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.251+0000 7fdebffff700 1 -- 172.21.15.78:0/914096635 learned_addr learned my addr 172.21.15.78:0/914096635 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:46.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.251+0000 7fdebffff700 1 -- 172.21.15.78:0/914096635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb00095f0 con 0x7fdec00ff690 2023-07-14T18:56:46.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.251+0000 7fdebffff700 1 --2- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 0x7fdec01959e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0006010 tx=0x7fdeb0004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:46.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.252+0000 7fdebd7fa700 1 -- 172.21.15.78:0/914096635 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb001d070 con 0x7fdec00ff690 2023-07-14T18:56:46.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.252+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdec0196120 con 0x7fdec00ff690 2023-07-14T18:56:46.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.252+0000 7fdebd7fa700 1 -- 172.21.15.78:0/914096635 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fdeb00044b0 con 0x7fdec00ff690 2023-07-14T18:56:46.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.252+0000 7fdebd7fa700 1 -- 172.21.15.78:0/914096635 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdeb0003840 con 0x7fdec00ff690 2023-07-14T18:56:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.252+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdec01964e0 con 0x7fdec00ff690 2023-07-14T18:56:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.252+0000 7fdebd7fa700 1 -- 172.21.15.78:0/914096635 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fdeb0003ac0 con 0x7fdec00ff690 2023-07-14T18:56:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.252+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdec0103bf0 con 0x7fdec00ff690 2023-07-14T18:56:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.253+0000 7fdebd7fa700 1 --2- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fdea80387b0 0x7fdea803ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.253+0000 7fdebd7fa700 1 -- 172.21.15.78:0/914096635 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdeb004c7e0 con 0x7fdec00ff690 2023-07-14T18:56:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.253+0000 7fdebf7fe700 1 --2- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fdea80387b0 0x7fdea803ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:46.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.253+0000 7fdebf7fe700 1 --2- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fdea80387b0 0x7fdea803ac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4006fd0 tx=0x7fdeb4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:46.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.257+0000 7fdebd7fa700 1 -- 172.21.15.78:0/914096635 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fdeb002a540 con 0x7fdec00ff690 2023-07-14T18:56:46.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:46.445+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}) v1 -- 0x7fdec00008d0 con 0x7fdea80387b0 2023-07-14T18:56:47.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:46 smithi078 ceph-mon[127425]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi088", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:48.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:47 smithi078 ceph-mon[127425]: Deploying cephadm binary to smithi088 2023-07-14T18:56:48.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.747+0000 7fdebd7fa700 1 -- 172.21.15.78:0/914096635 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fdec00008d0 con 0x7fdea80387b0 2023-07-14T18:56:48.753 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi088' with addr '172.21.15.88' 2023-07-14T18:56:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fdea80387b0 msgr2=0x7fdea803ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:48.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 --2- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fdea80387b0 0x7fdea803ac60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fdeb4006fd0 tx=0x7fdeb4006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 msgr2=0x7fdec01959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 --2- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 0x7fdec01959e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdeb0006010 tx=0x7fdeb0004ff0 comp rx=0 tx=0).stop 2023-07-14T18:56:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 shutdown_connections 2023-07-14T18:56:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 --2- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fdea80387b0 0x7fdea803ac60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 --2- 172.21.15.78:0/914096635 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fdec00ff690 0x7fdec01959e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 >> 172.21.15.78:0/914096635 conn(0x7fdec0075040 msgr2=0x7fdec00fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:48.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 shutdown_connections 2023-07-14T18:56:48.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:48.750+0000 7fdec62c8700 1 -- 172.21.15.78:0/914096635 wait complete. 2023-07-14T18:56:49.182 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T18:56:49.731 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:56:50.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:49 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:50.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:49 smithi078 ceph-mon[127425]: Added host smithi088 2023-07-14T18:56:50.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:49 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:50.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:49 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:50.833 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:50 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:50.833 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:50 smithi078 ceph-mon[127425]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:56:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 -- 172.21.15.78:0/2100065609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 msgr2=0x7f57fc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 --2- 172.21.15.78:0/2100065609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 0x7f57fc104a30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f57f0009910 tx=0x7f57f0009c20 comp rx=0 tx=0).stop 2023-07-14T18:56:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 -- 172.21.15.78:0/2100065609 shutdown_connections 2023-07-14T18:56:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 --2- 172.21.15.78:0/2100065609 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 0x7f57fc104a30 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 -- 172.21.15.78:0/2100065609 >> 172.21.15.78:0/2100065609 conn(0x7f57fc0743e0 msgr2=0x7f57fc0747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 -- 172.21.15.78:0/2100065609 shutdown_connections 2023-07-14T18:56:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 -- 172.21.15.78:0/2100065609 wait complete. 2023-07-14T18:56:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 Processor -- start 2023-07-14T18:56:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.023+0000 7f58030e1700 1 -- start start 2023-07-14T18:56:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f58030e1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 0x7f57fc195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f58030e1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc195d70 con 0x7f57fc104660 2023-07-14T18:56:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f5800e7d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 0x7f57fc195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f5800e7d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 0x7f57fc195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33510/0 (socket says 172.21.15.78:33510) 2023-07-14T18:56:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f5800e7d700 1 -- 172.21.15.78:0/1257398367 learned_addr learned my addr 172.21.15.78:0/1257398367 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f5800e7d700 1 -- 172.21.15.78:0/1257398367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f00095f0 con 0x7f57fc104660 2023-07-14T18:56:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f5800e7d700 1 --2- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 0x7f57fc195830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f57f0004f40 tx=0x7f57f0005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f57f9ffb700 1 -- 172.21.15.78:0/1257398367 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57f001d070 con 0x7f57fc104660 2023-07-14T18:56:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc195fd0 con 0x7f57fc104660 2023-07-14T18:56:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f57f9ffb700 1 -- 172.21.15.78:0/1257398367 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f57f00044b0 con 0x7f57fc104660 2023-07-14T18:56:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f57f9ffb700 1 -- 172.21.15.78:0/1257398367 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f57f0003810 con 0x7f57fc104660 2023-07-14T18:56:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.024+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc1964d0 con 0x7f57fc104660 2023-07-14T18:56:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.025+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57fc103900 con 0x7f57fc104660 2023-07-14T18:56:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.026+0000 7f57f9ffb700 1 -- 172.21.15.78:0/1257398367 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f57f000b960 con 0x7f57fc104660 2023-07-14T18:56:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.026+0000 7f57f9ffb700 1 --2- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f57ec0387f0 0x7f57ec03aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.026+0000 7f57f9ffb700 1 -- 172.21.15.78:0/1257398367 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f57f004cbd0 con 0x7f57fc104660 2023-07-14T18:56:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.026+0000 7f57fbfff700 1 --2- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f57ec0387f0 0x7f57ec03aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.026+0000 7f57fbfff700 1 --2- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f57ec0387f0 0x7f57ec03aca0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f57e8006fd0 tx=0x7f57e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.030+0000 7f57f9ffb700 1 -- 172.21.15.78:0/1257398367 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f57f00042e0 con 0x7f57fc104660 2023-07-14T18:56:51.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.221+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57fc067f60 con 0x7f57ec0387f0 2023-07-14T18:56:51.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.222+0000 7f57f9ffb700 1 -- 172.21.15.78:0/1257398367 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7f57fc067f60 con 0x7f57ec0387f0 2023-07-14T18:56:51.228 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:51.228 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}] 2023-07-14T18:56:51.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.224+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f57ec0387f0 msgr2=0x7f57ec03aca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:51.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.224+0000 7f58030e1700 1 --2- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f57ec0387f0 0x7f57ec03aca0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f57e8006fd0 tx=0x7f57e8006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:51.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.224+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 msgr2=0x7f57fc195830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:51.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.224+0000 7f58030e1700 1 --2- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 0x7f57fc195830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f57f0004f40 tx=0x7f57f0005020 comp rx=0 tx=0).stop 2023-07-14T18:56:51.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.224+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 shutdown_connections 2023-07-14T18:56:51.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.224+0000 7f58030e1700 1 --2- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f57ec0387f0 0x7f57ec03aca0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:51.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.224+0000 7f58030e1700 1 --2- 172.21.15.78:0/1257398367 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f57fc104660 0x7f57fc195830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:51.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.224+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 >> 172.21.15.78:0/1257398367 conn(0x7f57fc0743e0 msgr2=0x7f57fc105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:51.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.225+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 shutdown_connections 2023-07-14T18:56:51.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:51.225+0000 7f58030e1700 1 -- 172.21.15.78:0/1257398367 wait complete. 2023-07-14T18:56:51.690 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi102 2023-07-14T18:56:51.690 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:56:51.690 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/etc/ceph/ceph.conf 2023-07-14T18:56:51.711 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:56:51.711 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:56:51.776 INFO:tasks.cephadm:Adding host smithi102 to orchestrator... 2023-07-14T18:56:51.777 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch host add smithi102 2023-07-14T18:56:52.330 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:56:52.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:52.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:52 smithi078 ceph-mon[127425]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:56:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.534+0000 7fce1a1c8700 1 -- 172.21.15.78:0/1726075537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 msgr2=0x7fce140ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.534+0000 7fce1a1c8700 1 --2- 172.21.15.78:0/1726075537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 0x7fce140ffa60 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fce04009910 tx=0x7fce04009c20 comp rx=0 tx=0).stop 2023-07-14T18:56:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.535+0000 7fce1a1c8700 1 -- 172.21.15.78:0/1726075537 shutdown_connections 2023-07-14T18:56:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.535+0000 7fce1a1c8700 1 --2- 172.21.15.78:0/1726075537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 0x7fce140ffa60 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.535+0000 7fce1a1c8700 1 -- 172.21.15.78:0/1726075537 >> 172.21.15.78:0/1726075537 conn(0x7fce14075040 msgr2=0x7fce14075440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.535+0000 7fce1a1c8700 1 -- 172.21.15.78:0/1726075537 shutdown_connections 2023-07-14T18:56:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.535+0000 7fce1a1c8700 1 -- 172.21.15.78:0/1726075537 wait complete. 2023-07-14T18:56:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.535+0000 7fce1a1c8700 1 Processor -- start 2023-07-14T18:56:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce1a1c8700 1 -- start start 2023-07-14T18:56:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce1a1c8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 0x7fce1419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce1a1c8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce1419ae70 con 0x7fce140ff690 2023-07-14T18:56:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 0x7fce1419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce137fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 0x7fce1419a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33516/0 (socket says 172.21.15.78:33516) 2023-07-14T18:56:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce137fe700 1 -- 172.21.15.78:0/2610509168 learned_addr learned my addr 172.21.15.78:0/2610509168 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce137fe700 1 -- 172.21.15.78:0/2610509168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce040095f0 con 0x7fce140ff690 2023-07-14T18:56:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce137fe700 1 --2- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 0x7fce1419a930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fce04006010 tx=0x7fce04004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce10ff9700 1 -- 172.21.15.78:0/2610509168 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce0401d070 con 0x7fce140ff690 2023-07-14T18:56:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce1419b070 con 0x7fce140ff690 2023-07-14T18:56:53.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce10ff9700 1 -- 172.21.15.78:0/2610509168 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fce040044b0 con 0x7fce140ff690 2023-07-14T18:56:53.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce10ff9700 1 -- 172.21.15.78:0/2610509168 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fce04003840 con 0x7fce140ff690 2023-07-14T18:56:53.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.536+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce1419b430 con 0x7fce140ff690 2023-07-14T18:56:53.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.537+0000 7fce10ff9700 1 -- 172.21.15.78:0/2610509168 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fce04003ac0 con 0x7fce140ff690 2023-07-14T18:56:53.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.537+0000 7fce10ff9700 1 --2- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdfc0387b0 0x7fcdfc03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:53.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.537+0000 7fce10ff9700 1 -- 172.21.15.78:0/2610509168 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fce0404c7b0 con 0x7fce140ff690 2023-07-14T18:56:53.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.537+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce14061c70 con 0x7fce140ff690 2023-07-14T18:56:53.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.538+0000 7fce12ffd700 1 --2- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdfc0387b0 0x7fcdfc03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:53.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.538+0000 7fce12ffd700 1 --2- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdfc0387b0 0x7fcdfc03ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fce08006fd0 tx=0x7fce08006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:53.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.542+0000 7fce10ff9700 1 -- 172.21.15.78:0/2610509168 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fce04020b60 con 0x7fce140ff690 2023-07-14T18:56:53.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:53 smithi078 ceph-mon[127425]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:56:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:53.731+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}) v1 -- 0x7fce140fd6d0 con 0x7fcdfc0387b0 2023-07-14T18:56:55.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:56:55.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:55.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:55.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:55.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:55.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:56:55.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:56:55.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:56:55.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T18:56:55.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:56:55.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:56:55.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:56:55.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: Deploying cephadm binary to smithi102 2023-07-14T18:56:55.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:55.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:55.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:56.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.107+0000 7fce10ff9700 1 -- 172.21.15.78:0/2610509168 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fce140fd6d0 con 0x7fcdfc0387b0 2023-07-14T18:56:56.113 INFO:teuthology.orchestra.run.smithi078.stdout:Added host 'smithi102' with addr '172.21.15.102' 2023-07-14T18:56:56.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdfc0387b0 msgr2=0x7fcdfc03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:56.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 --2- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdfc0387b0 0x7fcdfc03ac60 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fce08006fd0 tx=0x7fce08006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:56.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 msgr2=0x7fce1419a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:56.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 --2- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 0x7fce1419a930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fce04006010 tx=0x7fce04004ff0 comp rx=0 tx=0).stop 2023-07-14T18:56:56.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 shutdown_connections 2023-07-14T18:56:56.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 --2- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcdfc0387b0 0x7fcdfc03ac60 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:56.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 --2- 172.21.15.78:0/2610509168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce140ff690 0x7fce1419a930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:56.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 >> 172.21.15.78:0/2610509168 conn(0x7fce14075040 msgr2=0x7fce140fcfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:56.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 shutdown_connections 2023-07-14T18:56:56.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:56.110+0000 7fce1a1c8700 1 -- 172.21.15.78:0/2610509168 wait complete. 2023-07-14T18:56:56.142 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:55 smithi078 ceph-mon[127425]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:56:56.576 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch host ls --format=json 2023-07-14T18:56:57.127 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:56:57.371 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:57 smithi078 ceph-mon[127425]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:56:57.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:57 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:57.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:57 smithi078 ceph-mon[127425]: Added host smithi102 2023-07-14T18:56:57.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:57 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:56:57.372 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:57 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.347+0000 7f82a13ae700 1 -- 172.21.15.78:0/2183792344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 msgr2=0x7f829c105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.347+0000 7f82a13ae700 1 --2- 172.21.15.78:0/2183792344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 0x7f829c105af0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f828c009910 tx=0x7f828c009c20 comp rx=0 tx=0).stop 2023-07-14T18:56:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.347+0000 7f82a13ae700 1 -- 172.21.15.78:0/2183792344 shutdown_connections 2023-07-14T18:56:58.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.347+0000 7f82a13ae700 1 --2- 172.21.15.78:0/2183792344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 0x7f829c105af0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.347+0000 7f82a13ae700 1 -- 172.21.15.78:0/2183792344 >> 172.21.15.78:0/2183792344 conn(0x7f829c0fb260 msgr2=0x7f829c0fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.347+0000 7f82a13ae700 1 -- 172.21.15.78:0/2183792344 shutdown_connections 2023-07-14T18:56:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.347+0000 7f82a13ae700 1 -- 172.21.15.78:0/2183792344 wait complete. 2023-07-14T18:56:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.348+0000 7f82a13ae700 1 Processor -- start 2023-07-14T18:56:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.348+0000 7f82a13ae700 1 -- start start 2023-07-14T18:56:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.348+0000 7f82a13ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 0x7f829c19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.348+0000 7f82a13ae700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f829c19ade0 con 0x7f829c105720 2023-07-14T18:56:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.348+0000 7f829affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 0x7f829c19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f829affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 0x7f829c19a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33542/0 (socket says 172.21.15.78:33542) 2023-07-14T18:56:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f829affd700 1 -- 172.21.15.78:0/3146825603 learned_addr learned my addr 172.21.15.78:0/3146825603 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:56:58.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f829affd700 1 -- 172.21.15.78:0/3146825603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f828c0095f0 con 0x7f829c105720 2023-07-14T18:56:58.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f829affd700 1 --2- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 0x7f829c19a8a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f828c005950 tx=0x7f828c004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f8283fff700 1 -- 172.21.15.78:0/3146825603 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f828c01d070 con 0x7f829c105720 2023-07-14T18:56:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f829c19afe0 con 0x7f829c105720 2023-07-14T18:56:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f8283fff700 1 -- 172.21.15.78:0/3146825603 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f828c00bdf0 con 0x7f829c105720 2023-07-14T18:56:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f8283fff700 1 -- 172.21.15.78:0/3146825603 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f828c022c50 con 0x7f829c105720 2023-07-14T18:56:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.349+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f829c194920 con 0x7f829c105720 2023-07-14T18:56:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.350+0000 7f8283fff700 1 -- 172.21.15.78:0/3146825603 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f828c022470 con 0x7f829c105720 2023-07-14T18:56:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.350+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f829c061c70 con 0x7f829c105720 2023-07-14T18:56:58.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.350+0000 7f8283fff700 1 --2- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82840387b0 0x7f828403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:56:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.350+0000 7f8283fff700 1 -- 172.21.15.78:0/3146825603 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f828c04ca00 con 0x7f829c105720 2023-07-14T18:56:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.350+0000 7f829a7fc700 1 --2- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82840387b0 0x7f828403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:56:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.351+0000 7f829a7fc700 1 --2- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82840387b0 0x7f828403ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8290006fd0 tx=0x7f8290006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:56:58.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.355+0000 7f8283fff700 1 -- 172.21.15.78:0/3146825603 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f828c02bba0 con 0x7f829c105720 2023-07-14T18:56:58.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:58 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:56:58.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.548+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f829c195350 con 0x7f82840387b0 2023-07-14T18:56:58.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.549+0000 7f8283fff700 1 -- 172.21.15.78:0/3146825603 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+239 (secure 0 0 0) 0x7f829c195350 con 0x7f82840387b0 2023-07-14T18:56:58.554 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T18:56:58.554 INFO:teuthology.orchestra.run.smithi078.stdout:[{"addr": "172.21.15.78", "hostname": "smithi078", "labels": [], "status": ""}, {"addr": "172.21.15.88", "hostname": "smithi088", "labels": [], "status": ""}, {"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}] 2023-07-14T18:56:58.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82840387b0 msgr2=0x7f828403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:58.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 --2- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82840387b0 0x7f828403ac60 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f8290006fd0 tx=0x7f8290006e40 comp rx=0 tx=0).stop 2023-07-14T18:56:58.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 msgr2=0x7f829c19a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:56:58.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 --2- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 0x7f829c19a8a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f828c005950 tx=0x7f828c004ff0 comp rx=0 tx=0).stop 2023-07-14T18:56:58.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 shutdown_connections 2023-07-14T18:56:58.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 --2- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82840387b0 0x7f828403ac60 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:58.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 --2- 172.21.15.78:0/3146825603 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f829c105720 0x7f829c19a8a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:56:58.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 >> 172.21.15.78:0/3146825603 conn(0x7f829c0fb260 msgr2=0x7f829c0fbec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:56:58.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 shutdown_connections 2023-07-14T18:56:58.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:56:58.551+0000 7f82a13ae700 1 -- 172.21.15.78:0/3146825603 wait complete. 2023-07-14T18:56:58.982 INFO:tasks.cephadm:Setting crush tunables to default 2023-07-14T18:56:58.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd crush tunables default 2023-07-14T18:56:59.532 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:56:59.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:59 smithi078 ceph-mon[127425]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:56:59.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:56:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:00.315 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:00 smithi078 ceph-mon[127425]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:57:00.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.743+0000 7fe766748700 1 -- 172.21.15.78:0/4173515014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 msgr2=0x7fe7600ffa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:00.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.743+0000 7fe766748700 1 --2- 172.21.15.78:0/4173515014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 0x7fe7600ffa60 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe754009910 tx=0x7fe754009c20 comp rx=0 tx=0).stop 2023-07-14T18:57:00.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.743+0000 7fe766748700 1 -- 172.21.15.78:0/4173515014 shutdown_connections 2023-07-14T18:57:00.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.743+0000 7fe766748700 1 --2- 172.21.15.78:0/4173515014 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 0x7fe7600ffa60 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:00.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.743+0000 7fe766748700 1 -- 172.21.15.78:0/4173515014 >> 172.21.15.78:0/4173515014 conn(0x7fe760075040 msgr2=0x7fe760075440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:00.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.743+0000 7fe766748700 1 -- 172.21.15.78:0/4173515014 shutdown_connections 2023-07-14T18:57:00.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.743+0000 7fe766748700 1 -- 172.21.15.78:0/4173515014 wait complete. 2023-07-14T18:57:00.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.743+0000 7fe766748700 1 Processor -- start 2023-07-14T18:57:00.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.744+0000 7fe766748700 1 -- start start 2023-07-14T18:57:00.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.744+0000 7fe766748700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 0x7fe7601959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:00.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.744+0000 7fe766748700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe760195f20 con 0x7fe7600ff690 2023-07-14T18:57:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.744+0000 7fe75ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 0x7fe7601959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.744+0000 7fe75ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 0x7fe7601959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33548/0 (socket says 172.21.15.78:33548) 2023-07-14T18:57:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.744+0000 7fe75ffff700 1 -- 172.21.15.78:0/1181204817 learned_addr learned my addr 172.21.15.78:0/1181204817 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:57:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.744+0000 7fe75ffff700 1 -- 172.21.15.78:0/1181204817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7540095f0 con 0x7fe7600ff690 2023-07-14T18:57:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.744+0000 7fe75ffff700 1 --2- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 0x7fe7601959e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe754006010 tx=0x7fe754004ff0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.745+0000 7fe75d7fa700 1 -- 172.21.15.78:0/1181204817 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe75401d070 con 0x7fe7600ff690 2023-07-14T18:57:00.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.745+0000 7fe75d7fa700 1 -- 172.21.15.78:0/1181204817 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe7540044b0 con 0x7fe7600ff690 2023-07-14T18:57:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.745+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe760196120 con 0x7fe7600ff690 2023-07-14T18:57:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.745+0000 7fe75d7fa700 1 -- 172.21.15.78:0/1181204817 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe754003840 con 0x7fe7600ff690 2023-07-14T18:57:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.745+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7601964e0 con 0x7fe7600ff690 2023-07-14T18:57:00.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.745+0000 7fe75d7fa700 1 -- 172.21.15.78:0/1181204817 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fe754003a70 con 0x7fe7600ff690 2023-07-14T18:57:00.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.746+0000 7fe75d7fa700 1 --2- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe748038800 0x7fe74803acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:00.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.746+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe760103bf0 con 0x7fe7600ff690 2023-07-14T18:57:00.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.746+0000 7fe75d7fa700 1 -- 172.21.15.78:0/1181204817 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe75404cab0 con 0x7fe7600ff690 2023-07-14T18:57:00.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.746+0000 7fe75f7fe700 1 --2- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe748038800 0x7fe74803acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:00.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.746+0000 7fe75f7fe700 1 --2- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe748038800 0x7fe74803acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe750006fd0 tx=0x7fe750006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:00.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.750+0000 7fe75d7fa700 1 -- 172.21.15.78:0/1181204817 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe75402a770 con 0x7fe7600ff690 2023-07-14T18:57:00.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:00.944+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe7600fe910 con 0x7fe7600ff690 2023-07-14T18:57:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:57:01.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:01.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:01.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:01 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1181204817' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:57:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.872+0000 7fe75d7fa700 1 -- 172.21.15.78:0/1181204817 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe75402a360 con 0x7fe7600ff690 2023-07-14T18:57:01.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe748038800 msgr2=0x7fe74803acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:01.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 --2- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe748038800 0x7fe74803acb0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe750006fd0 tx=0x7fe750006e40 comp rx=0 tx=0).stop 2023-07-14T18:57:01.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 msgr2=0x7fe7601959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:01.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 --2- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 0x7fe7601959e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe754006010 tx=0x7fe754004ff0 comp rx=0 tx=0).stop 2023-07-14T18:57:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 shutdown_connections 2023-07-14T18:57:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 --2- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe748038800 0x7fe74803acb0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 --2- 172.21.15.78:0/1181204817 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7600ff690 0x7fe7601959e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 >> 172.21.15.78:0/1181204817 conn(0x7fe760075040 msgr2=0x7fe7600fc270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 shutdown_connections 2023-07-14T18:57:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:01.875+0000 7fe766748700 1 -- 172.21.15.78:0/1181204817 wait complete. 2023-07-14T18:57:01.882 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2023-07-14T18:57:02.378 INFO:tasks.cephadm:Adding mon.a on smithi078 2023-07-14T18:57:02.378 INFO:tasks.cephadm:Adding mon.b on smithi088 2023-07-14T18:57:02.378 INFO:tasks.cephadm:Adding mon.c on smithi102 2023-07-14T18:57:02.379 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch apply mon '3;smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c' 2023-07-14T18:57:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:02 smithi078 ceph-mon[127425]: Updating smithi102:/etc/ceph/ceph.conf 2023-07-14T18:57:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:02 smithi078 ceph-mon[127425]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:02 smithi078 ceph-mon[127425]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:57:02.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:02.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:02.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:02.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:02 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1181204817' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:57:02.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:02 smithi078 ceph-mon[127425]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:02.963 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:03.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:03 smithi078 ceph-mon[127425]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:57:03.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:03 smithi078 ceph-mon[127425]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:04.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 -- 172.21.15.102:0/883034511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 msgr2=0x7fb494104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:04.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 --2- 172.21.15.102:0/883034511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 0x7fb494104a30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb484009910 tx=0x7fb484009c20 comp rx=0 tx=0).stop 2023-07-14T18:57:04.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 -- 172.21.15.102:0/883034511 shutdown_connections 2023-07-14T18:57:04.067 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 --2- 172.21.15.102:0/883034511 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 0x7fb494104a30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:04.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 -- 172.21.15.102:0/883034511 >> 172.21.15.102:0/883034511 conn(0x7fb4940743e0 msgr2=0x7fb4940747e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:04.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 -- 172.21.15.102:0/883034511 shutdown_connections 2023-07-14T18:57:04.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 -- 172.21.15.102:0/883034511 wait complete. 2023-07-14T18:57:04.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 Processor -- start 2023-07-14T18:57:04.068 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.061+0000 7fb49a9b3700 1 -- start start 2023-07-14T18:57:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.062+0000 7fb49a9b3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 0x7fb494195830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.062+0000 7fb49a9b3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb494195d70 con 0x7fb494104660 2023-07-14T18:57:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.062+0000 7fb493fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 0x7fb494195830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.062+0000 7fb493fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 0x7fb494195830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.102:55632/0 (socket says 172.21.15.102:55632) 2023-07-14T18:57:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.062+0000 7fb493fff700 1 -- 172.21.15.102:0/1087344912 learned_addr learned my addr 172.21.15.102:0/1087344912 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T18:57:04.070 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.062+0000 7fb493fff700 1 -- 172.21.15.102:0/1087344912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4840095f0 con 0x7fb494104660 2023-07-14T18:57:04.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.063+0000 7fb493fff700 1 --2- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 0x7fb494195830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb484004f40 tx=0x7fb484005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:04.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.063+0000 7fb4917fa700 1 -- 172.21.15.102:0/1087344912 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb48401d070 con 0x7fb494104660 2023-07-14T18:57:04.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.063+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb494195fd0 con 0x7fb494104660 2023-07-14T18:57:04.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.063+0000 7fb4917fa700 1 -- 172.21.15.102:0/1087344912 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb4840044b0 con 0x7fb494104660 2023-07-14T18:57:04.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.063+0000 7fb4917fa700 1 -- 172.21.15.102:0/1087344912 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4840037f0 con 0x7fb494104660 2023-07-14T18:57:04.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.063+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4941964d0 con 0x7fb494104660 2023-07-14T18:57:04.071 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.064+0000 7fb4917fa700 1 -- 172.21.15.102:0/1087344912 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fb484003a70 con 0x7fb494104660 2023-07-14T18:57:04.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.064+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb494103900 con 0x7fb494104660 2023-07-14T18:57:04.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.064+0000 7fb4917fa700 1 --2- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb47c0387b0 0x7fb47c03ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:04.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.064+0000 7fb4917fa700 1 -- 172.21.15.102:0/1087344912 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb48404c710 con 0x7fb494104660 2023-07-14T18:57:04.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.064+0000 7fb4937fe700 1 --2- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb47c0387b0 0x7fb47c03ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:04.072 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.065+0000 7fb4937fe700 1 --2- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb47c0387b0 0x7fb47c03ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb488006fd0 tx=0x7fb488006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:04.074 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.069+0000 7fb4917fa700 1 -- 172.21.15.102:0/1087344912 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb4840042e0 con 0x7fb494104660 2023-07-14T18:57:04.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.262+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c", "target": ["mon-mgr", ""]}) v1 -- 0x7fb494068020 con 0x7fb47c0387b0 2023-07-14T18:57:04.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.271+0000 7fb4917fa700 1 -- 172.21.15.102:0/1087344912 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fb488009e20 con 0x7fb47c0387b0 2023-07-14T18:57:04.276 INFO:teuthology.orchestra.run.smithi102.stdout:Scheduled mon update... 2023-07-14T18:57:04.278 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb47c0387b0 msgr2=0x7fb47c03ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:04.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 --2- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb47c0387b0 0x7fb47c03ac60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb488006fd0 tx=0x7fb488006e40 comp rx=0 tx=0).stop 2023-07-14T18:57:04.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 msgr2=0x7fb494195830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:04.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 --2- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 0x7fb494195830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb484004f40 tx=0x7fb484005020 comp rx=0 tx=0).stop 2023-07-14T18:57:04.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 shutdown_connections 2023-07-14T18:57:04.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 --2- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb47c0387b0 0x7fb47c03ac60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:04.279 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 --2- 172.21.15.102:0/1087344912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb494104660 0x7fb494195830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:04.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 >> 172.21.15.102:0/1087344912 conn(0x7fb4940743e0 msgr2=0x7fb494105c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:04.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 shutdown_connections 2023-07-14T18:57:04.280 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:04.273+0000 7fb49a9b3700 1 -- 172.21.15.102:0/1087344912 wait complete. 2023-07-14T18:57:04.625 DEBUG:teuthology.orchestra.run.smithi088:mon.b> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.b.service 2023-07-14T18:57:04.628 DEBUG:teuthology.orchestra.run.smithi102:mon.c> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.c.service 2023-07-14T18:57:04.633 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:57:04.633 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:57:04.661 INFO:journalctl@ceph.mon.c.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T18:57:04.664 INFO:journalctl@ceph.mon.b.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T18:57:05.296 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:05.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:05.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:05.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c;count:3 2023-07-14T18:57:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:05 smithi078 ceph-mon[127425]: Deploying daemon mon.c on smithi102 2023-07-14T18:57:06.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:06 smithi078 ceph-mon[127425]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.839+0000 7f9d80dc1700 1 -- 172.21.15.102:0/2763619424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 msgr2=0x7f9d7c0fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.839+0000 7f9d80dc1700 1 --2- 172.21.15.102:0/2763619424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 0x7f9d7c0fe840 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009910 tx=0x7f9d6c009c20 comp rx=0 tx=0).stop 2023-07-14T18:57:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.840+0000 7f9d80dc1700 1 -- 172.21.15.102:0/2763619424 shutdown_connections 2023-07-14T18:57:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.840+0000 7f9d80dc1700 1 --2- 172.21.15.102:0/2763619424 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 0x7f9d7c0fe840 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:06.846 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.840+0000 7f9d80dc1700 1 -- 172.21.15.102:0/2763619424 >> 172.21.15.102:0/2763619424 conn(0x7f9d7c0fa140 msgr2=0x7f9d7c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.840+0000 7f9d80dc1700 1 -- 172.21.15.102:0/2763619424 shutdown_connections 2023-07-14T18:57:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.840+0000 7f9d80dc1700 1 -- 172.21.15.102:0/2763619424 wait complete. 2023-07-14T18:57:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.841+0000 7f9d80dc1700 1 Processor -- start 2023-07-14T18:57:06.847 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.841+0000 7f9d80dc1700 1 -- start start 2023-07-14T18:57:06.848 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.841+0000 7f9d80dc1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 0x7f9d7c196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:06.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.841+0000 7f9d80dc1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d7c197260 con 0x7f9d7c0fe470 2023-07-14T18:57:06.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.841+0000 7f9d7a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 0x7f9d7c196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:06.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.841+0000 7f9d7a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 0x7f9d7c196d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.102:55638/0 (socket says 172.21.15.102:55638) 2023-07-14T18:57:06.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.841+0000 7f9d7a59c700 1 -- 172.21.15.102:0/62612413 learned_addr learned my addr 172.21.15.102:0/62612413 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T18:57:06.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.841+0000 7f9d7a59c700 1 -- 172.21.15.102:0/62612413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c0095f0 con 0x7f9d7c0fe470 2023-07-14T18:57:06.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.842+0000 7f9d7a59c700 1 --2- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 0x7f9d7c196d20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c004f40 tx=0x7f9d6c005020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:06.849 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.842+0000 7f9d6b7fe700 1 -- 172.21.15.102:0/62612413 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d6c01d070 con 0x7f9d7c0fe470 2023-07-14T18:57:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.842+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d7c197460 con 0x7f9d7c0fe470 2023-07-14T18:57:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.842+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d7c197820 con 0x7f9d7c0fe470 2023-07-14T18:57:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.842+0000 7f9d6b7fe700 1 -- 172.21.15.102:0/62612413 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9d6c00bdf0 con 0x7f9d7c0fe470 2023-07-14T18:57:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.842+0000 7f9d6b7fe700 1 -- 172.21.15.102:0/62612413 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9d6c022c30 con 0x7f9d7c0fe470 2023-07-14T18:57:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.843+0000 7f9d6b7fe700 1 -- 172.21.15.102:0/62612413 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f9d6c00f460 con 0x7f9d7c0fe470 2023-07-14T18:57:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.843+0000 7f9d6b7fe700 1 --2- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9d640387b0 0x7f9d6403ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.843+0000 7f9d6b7fe700 1 -- 172.21.15.102:0/62612413 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f9d6c04c910 con 0x7f9d7c0fe470 2023-07-14T18:57:06.850 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.843+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5c005320 con 0x7f9d7c0fe470 2023-07-14T18:57:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.843+0000 7f9d79d9b700 1 --2- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9d640387b0 0x7f9d6403ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:06.851 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.844+0000 7f9d79d9b700 1 --2- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9d640387b0 0x7f9d6403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9d70006fd0 tx=0x7f9d70006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:06.853 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:06.848+0000 7f9d6b7fe700 1 -- 172.21.15.102:0/62612413 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9d6c022470 con 0x7f9d7c0fe470 2023-07-14T18:57:07.155 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.149+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d5c0059f0 con 0x7f9d7c0fe470 2023-07-14T18:57:07.156 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.151+0000 7f9d6b7fe700 1 -- 172.21.15.102:0/62612413 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+736 (secure 0 0 0) 0x7f9d6c02ab50 con 0x7f9d7c0fe470 2023-07-14T18:57:07.156 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:57:07.157 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":1,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:55:32.052335Z","created":"2023-07-14T18:55:32.052335Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-07-14T18:57:07.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9d640387b0 msgr2=0x7f9d6403ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:07.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 --2- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9d640387b0 0x7f9d6403ac60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f9d70006fd0 tx=0x7f9d70006e40 comp rx=0 tx=0).stop 2023-07-14T18:57:07.158 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 msgr2=0x7f9d7c196d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 --2- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 0x7f9d7c196d20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c004f40 tx=0x7f9d6c005020 comp rx=0 tx=0).stop 2023-07-14T18:57:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 shutdown_connections 2023-07-14T18:57:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 --2- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9d640387b0 0x7f9d6403ac60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 --2- 172.21.15.102:0/62612413 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d7c0fe470 0x7f9d7c196d20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 >> 172.21.15.102:0/62612413 conn(0x7f9d7c0fa140 msgr2=0x7f9d7c0fbb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 shutdown_connections 2023-07-14T18:57:07.159 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:07.153+0000 7f9d80dc1700 1 -- 172.21.15.102:0/62612413 wait complete. 2023-07-14T18:57:07.160 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 1 2023-07-14T18:57:07.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:07 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/62612413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:57:08.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 systemd[1]: Started Ceph mon.c for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T18:57:08.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).mds e1 new map 2023-07-14T18:57:08.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).mds e1 print_map 2023-07-14T18:57:08.133 INFO:journalctl@ceph.mon.c.smithi102.stdout: e1 2023-07-14T18:57:08.134 INFO:journalctl@ceph.mon.c.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:57:08.134 INFO:journalctl@ceph.mon.c.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:57:08.134 INFO:journalctl@ceph.mon.c.smithi102.stdout: legacy client fscid: -1 2023-07-14T18:57:08.134 INFO:journalctl@ceph.mon.c.smithi102.stdout: 2023-07-14T18:57:08.134 INFO:journalctl@ceph.mon.c.smithi102.stdout: No filesystems configured 2023-07-14T18:57:08.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T18:57:08.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T18:57:08.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T18:57:08.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T18:57:08.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T18:57:08.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T18:57:08.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:57:08.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:57:08.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:57:08.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:57:08.136 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:08.136 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:08.136 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.136 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.136 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:57:08.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:08.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:08.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T18:57:08.137 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:08.138 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:08.138 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:57:08.138 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Deploying cephadm binary to smithi102 2023-07-14T18:57:08.138 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.138 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.138 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.138 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:57:08.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:08.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Added host smithi102 2023-07-14T18:57:08.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:08.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:08.139 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:57:08.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:08.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.140 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:57:08.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:08.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:08.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1181204817' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:57:08.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Updating smithi102:/etc/ceph/ceph.conf 2023-07-14T18:57:08.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:08.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:57:08.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.141 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.142 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.142 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1181204817' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:57:08.142 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:08.142 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:57:08.142 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:08.142 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:08.142 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:08.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c;count:3 2023-07-14T18:57:08.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:08.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:08.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:08.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:08.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:08.143 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:08.144 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: Deploying daemon mon.c on smithi102 2023-07-14T18:57:08.144 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:08.144 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/62612413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:57:08.144 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T18:57:08.144 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:07 smithi102 ceph-mon[129280]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-07-14T18:57:08.627 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:57:08.628 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:57:09.185 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:57:13.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.232+0000 7fd4e7456700 1 -- 172.21.15.102:0/3720689132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 msgr2=0x7fd4cc005a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:13.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.232+0000 7fd4e7456700 1 --2- 172.21.15.102:0/3720689132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 0x7fd4cc005a40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00ffbd0 tx=0x7fd4d000fe60 comp rx=0 tx=0).stop 2023-07-14T18:57:13.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.232+0000 7fd4e7456700 1 -- 172.21.15.102:0/3720689132 shutdown_connections 2023-07-14T18:57:13.238 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.232+0000 7fd4e7456700 1 --2- 172.21.15.102:0/3720689132 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 0x7fd4cc005a40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:13.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.232+0000 7fd4e7456700 1 -- 172.21.15.102:0/3720689132 >> 172.21.15.102:0/3720689132 conn(0x7fd4e00fa260 msgr2=0x7fd4e00fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:13.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.232+0000 7fd4e7456700 1 -- 172.21.15.102:0/3720689132 shutdown_connections 2023-07-14T18:57:13.239 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.232+0000 7fd4e7456700 1 -- 172.21.15.102:0/3720689132 wait complete. 2023-07-14T18:57:13.240 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.233+0000 7fd4e7456700 1 Processor -- start 2023-07-14T18:57:13.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.233+0000 7fd4e7456700 1 -- start start 2023-07-14T18:57:13.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.233+0000 7fd4e7456700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 0x7fd4e0194cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:13.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.233+0000 7fd4e7456700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 0x7fd4e01985b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:13.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e7456700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e0195770 con 0x7fd4cc0035c0 2023-07-14T18:57:13.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e7456700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e01958b0 con 0x7fd4e01951f0 2023-07-14T18:57:13.241 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e49f1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 0x7fd4e01985b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:13.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e49f1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 0x7fd4e01985b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:51986/0 (socket says 172.21.15.102:51986) 2023-07-14T18:57:13.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e49f1700 1 -- 172.21.15.102:0/607165267 learned_addr learned my addr 172.21.15.102:0/607165267 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T18:57:13.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e51f2700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 0x7fd4e0194cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:13.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e49f1700 1 -- 172.21.15.102:0/607165267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 msgr2=0x7fd4e01985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T18:57:13.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e49f1700 1 -- 172.21.15.102:0/607165267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 msgr2=0x7fd4e01985b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:57:13.242 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e49f1700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 0x7fd4e01985b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:57:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e49f1700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 0x7fd4e01985b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:57:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e51f2700 1 -- 172.21.15.102:0/607165267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 msgr2=0x7fd4e01985b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:57:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e51f2700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 0x7fd4e01985b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e51f2700 1 -- 172.21.15.102:0/607165267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d000f8b0 con 0x7fd4cc0035c0 2023-07-14T18:57:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.234+0000 7fd4e51f2700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 0x7fd4e0194cb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00ffbd0 tx=0x7fd4d0013c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.235+0000 7fd4d67fc700 1 -- 172.21.15.102:0/607165267 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd4d0023070 con 0x7fd4cc0035c0 2023-07-14T18:57:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.235+0000 7fd4d67fc700 1 -- 172.21.15.102:0/607165267 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd4d00037c0 con 0x7fd4cc0035c0 2023-07-14T18:57:13.243 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.235+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e0198af0 con 0x7fd4cc0035c0 2023-07-14T18:57:13.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.235+0000 7fd4d67fc700 1 -- 172.21.15.102:0/607165267 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7fd4d001cb70 con 0x7fd4cc0035c0 2023-07-14T18:57:13.244 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.235+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e0199040 con 0x7fd4cc0035c0 2023-07-14T18:57:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.236+0000 7fd4d67fc700 1 -- 172.21.15.102:0/607165267 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fd4d001ccd0 con 0x7fd4cc0035c0 2023-07-14T18:57:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.236+0000 7fd4d67fc700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd4c40385f0 0x7fd4c403aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.236+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4e0061c70 con 0x7fd4cc0035c0 2023-07-14T18:57:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.236+0000 7fd4d67fc700 1 -- 172.21.15.102:0/607165267 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fd4d004d340 con 0x7fd4cc0035c0 2023-07-14T18:57:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.237+0000 7fd4e49f1700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd4c40385f0 0x7fd4c403aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:13.245 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.237+0000 7fd4e49f1700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd4c40385f0 0x7fd4c403aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009180 tx=0x7fd4dc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:13.247 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.242+0000 7fd4d67fc700 1 -- 172.21.15.102:0/607165267 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd4d002f950 con 0x7fd4cc0035c0 2023-07-14T18:57:13.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:57:13.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: mon.a calling monitor election 2023-07-14T18:57:13.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:13.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: mon.c calling monitor election 2023-07-14T18:57:13.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:13.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:13.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:57:13.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:13.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: fsmap 2023-07-14T18:57:13.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:13.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: mgrmap e13: x(active, since 42s) 2023-07-14T18:57:13.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: overall HEALTH_OK 2023-07-14T18:57:13.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:13.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:13.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:13.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:13.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:13 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:13.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.503+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4e0199370 con 0x7fd4cc0035c0 2023-07-14T18:57:13.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.503+0000 7fd4d67fc700 1 -- 172.21.15.102:0/607165267 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1006 (secure 0 0 0) 0x7fd4d002f360 con 0x7fd4cc0035c0 2023-07-14T18:57:13.509 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:57:13.510 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":2,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:57:07.958573Z","created":"2023-07-14T18:55:32.052335Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T18:57:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.505+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd4c40385f0 msgr2=0x7fd4c403aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.505+0000 7fd4e7456700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd4c40385f0 0x7fd4c403aaa0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009180 tx=0x7fd4dc008040 comp rx=0 tx=0).stop 2023-07-14T18:57:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.505+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 msgr2=0x7fd4e0194cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:13.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.505+0000 7fd4e7456700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 0x7fd4e0194cb0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd4e00ffbd0 tx=0x7fd4d0013c40 comp rx=0 tx=0).stop 2023-07-14T18:57:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.506+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 shutdown_connections 2023-07-14T18:57:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.506+0000 7fd4e7456700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd4c40385f0 0x7fd4c403aaa0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.506+0000 7fd4e7456700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd4cc0035c0 0x7fd4e0194cb0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.506+0000 7fd4e7456700 1 --2- 172.21.15.102:0/607165267 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e01951f0 0x7fd4e01985b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.506+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 >> 172.21.15.102:0/607165267 conn(0x7fd4e00fa260 msgr2=0x7fd4e0102490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.506+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 shutdown_connections 2023-07-14T18:57:13.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:13.506+0000 7fd4e7456700 1 -- 172.21.15.102:0/607165267 wait complete. 2023-07-14T18:57:13.513 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 2 2023-07-14T18:57:13.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:57:13.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: mon.a calling monitor election 2023-07-14T18:57:13.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:13.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: mon.c calling monitor election 2023-07-14T18:57:13.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:13.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:13.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:13.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:57:13.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:13.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: fsmap 2023-07-14T18:57:13.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:13.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: mgrmap e13: x(active, since 42s) 2023-07-14T18:57:13.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: overall HEALTH_OK 2023-07-14T18:57:13.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:13.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:13.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:13.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:13.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:13 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:14.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:14 smithi102 ceph-mon[129280]: Deploying daemon mon.b on smithi088 2023-07-14T18:57:14.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:14 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/607165267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:57:14.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:14.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:14 smithi078 ceph-mon[127425]: Deploying daemon mon.b on smithi088 2023-07-14T18:57:14.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:14 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/607165267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:57:14.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:14.947 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-07-14T18:57:14.947 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph mon dump -f json 2023-07-14T18:57:15.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:15 smithi102 ceph-mon[129280]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:15.509 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:57:15.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:15 smithi078 ceph-mon[127425]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:15.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:15 smithi088 systemd[1]: Starting Ceph mon.b for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T18:57:16.260 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:15 smithi088 podman[128934]: 2023-07-14T18:57:16.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: set uid:gid to 167:167 (ceph:ceph) 2023-07-14T18:57:16.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev), process ceph-mon, pid 7 2023-07-14T18:57:16.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pidfile_write: ignore empty --pid-file 2023-07-14T18:57:16.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 bash[128934]: b92907992a5ad8094de297055f6d9ccf4e1e236d943b22d6b86975a91aff5e19 2023-07-14T18:57:16.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: load: jerasure load: lrc 2023-07-14T18:57:16.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: RocksDB version: 7.9.2 2023-07-14T18:57:16.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Git sha 0 2023-07-14T18:57:16.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Compile date 2023-07-14 14:45:42 2023-07-14T18:57:16.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: DB SUMMARY 2023-07-14T18:57:16.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: DB Session ID: TXIRROTI1KTGNBOO2PQQ 2023-07-14T18:57:16.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: CURRENT file: CURRENT 2023-07-14T18:57:16.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: IDENTITY file: IDENTITY 2023-07-14T18:57:16.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-07-14T18:57:16.263 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2023-07-14T18:57:16.263 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2023-07-14T18:57:16.263 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.error_if_exists: 0 2023-07-14T18:57:16.263 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.create_if_missing: 0 2023-07-14T18:57:16.263 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.paranoid_checks: 1 2023-07-14T18:57:16.263 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.flush_verify_memtable_count: 1 2023-07-14T18:57:16.263 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-07-14T18:57:16.263 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-07-14T18:57:16.264 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.env: 0x5567c05c4c60 2023-07-14T18:57:16.264 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.fs: PosixFileSystem 2023-07-14T18:57:16.264 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.info_log: 0x5567c25f72a0 2023-07-14T18:57:16.264 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_file_opening_threads: 16 2023-07-14T18:57:16.264 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.statistics: (nil) 2023-07-14T18:57:16.264 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.use_fsync: 0 2023-07-14T18:57:16.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_log_file_size: 0 2023-07-14T18:57:16.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-07-14T18:57:16.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.log_file_time_to_roll: 0 2023-07-14T18:57:16.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.keep_log_file_num: 1000 2023-07-14T18:57:16.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.recycle_log_file_num: 0 2023-07-14T18:57:16.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.allow_fallocate: 1 2023-07-14T18:57:16.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.allow_mmap_reads: 0 2023-07-14T18:57:16.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.allow_mmap_writes: 0 2023-07-14T18:57:16.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.use_direct_reads: 0 2023-07-14T18:57:16.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-07-14T18:57:16.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.create_missing_column_families: 0 2023-07-14T18:57:16.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.db_log_dir: 2023-07-14T18:57:16.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.wal_dir: 2023-07-14T18:57:16.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.table_cache_numshardbits: 6 2023-07-14T18:57:16.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.WAL_ttl_seconds: 0 2023-07-14T18:57:16.266 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.WAL_size_limit_MB: 0 2023-07-14T18:57:16.267 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-07-14T18:57:16.267 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-07-14T18:57:16.267 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.is_fd_close_on_exec: 1 2023-07-14T18:57:16.267 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.advise_random_on_open: 1 2023-07-14T18:57:16.267 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.db_write_buffer_size: 0 2023-07-14T18:57:16.267 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.write_buffer_manager: 0x5567c25183c0 2023-07-14T18:57:16.267 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-07-14T18:57:16.267 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-07-14T18:57:16.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.use_adaptive_mutex: 0 2023-07-14T18:57:16.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.rate_limiter: (nil) 2023-07-14T18:57:16.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-07-14T18:57:16.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.wal_recovery_mode: 2 2023-07-14T18:57:16.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.enable_thread_tracking: 0 2023-07-14T18:57:16.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.enable_pipelined_write: 0 2023-07-14T18:57:16.268 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.unordered_write: 0 2023-07-14T18:57:16.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-07-14T18:57:16.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-07-14T18:57:16.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-07-14T18:57:16.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-07-14T18:57:16.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.row_cache: None 2023-07-14T18:57:16.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.wal_filter: None 2023-07-14T18:57:16.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-07-14T18:57:16.269 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.allow_ingest_behind: 0 2023-07-14T18:57:16.270 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.two_write_queues: 0 2023-07-14T18:57:16.270 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.manual_wal_flush: 0 2023-07-14T18:57:16.270 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.wal_compression: 0 2023-07-14T18:57:16.270 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.atomic_flush: 0 2023-07-14T18:57:16.270 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-07-14T18:57:16.270 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.persist_stats_to_disk: 0 2023-07-14T18:57:16.270 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.write_dbid_to_manifest: 0 2023-07-14T18:57:16.270 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.log_readahead_size: 0 2023-07-14T18:57:16.271 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-07-14T18:57:16.271 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.best_efforts_recovery: 0 2023-07-14T18:57:16.271 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-07-14T18:57:16.271 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-07-14T18:57:16.272 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.allow_data_in_errors: 0 2023-07-14T18:57:16.273 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.db_host_id: __hostname__ 2023-07-14T18:57:16.273 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.enforce_single_del_contracts: true 2023-07-14T18:57:16.273 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_background_jobs: 2 2023-07-14T18:57:16.273 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_background_compactions: -1 2023-07-14T18:57:16.273 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_subcompactions: 1 2023-07-14T18:57:16.273 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-07-14T18:57:16.274 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-07-14T18:57:16.274 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.delayed_write_rate : 16777216 2023-07-14T18:57:16.274 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_total_wal_size: 0 2023-07-14T18:57:16.274 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-07-14T18:57:16.274 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.stats_dump_period_sec: 600 2023-07-14T18:57:16.274 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.stats_persist_period_sec: 600 2023-07-14T18:57:16.274 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-07-14T18:57:16.274 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_open_files: -1 2023-07-14T18:57:16.275 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bytes_per_sync: 0 2023-07-14T18:57:16.275 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.wal_bytes_per_sync: 0 2023-07-14T18:57:16.275 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.strict_bytes_per_sync: 0 2023-07-14T18:57:16.275 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_readahead_size: 0 2023-07-14T18:57:16.275 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_background_flushes: -1 2023-07-14T18:57:16.275 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Compression algorithms supported: 2023-07-14T18:57:16.275 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-07-14T18:57:16.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: kZSTD supported: 0 2023-07-14T18:57:16.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: kXpressCompression supported: 0 2023-07-14T18:57:16.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: kLZ4HCCompression supported: 1 2023-07-14T18:57:16.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: kZlibCompression supported: 1 2023-07-14T18:57:16.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: kSnappyCompression supported: 1 2023-07-14T18:57:16.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: kLZ4Compression supported: 1 2023-07-14T18:57:16.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: kBZip2Compression supported: 0 2023-07-14T18:57:16.276 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Fast CRC32 supported: Supported on x86 2023-07-14T18:57:16.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: DMutex implementation: pthread_mutex_t 2023-07-14T18:57:16.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2023-07-14T18:57:16.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-07-14T18:57:16.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-07-14T18:57:16.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.merge_operator: 2023-07-14T18:57:16.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_filter: None 2023-07-14T18:57:16.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_filter_factory: None 2023-07-14T18:57:16.277 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.sst_partitioner_factory: None 2023-07-14T18:57:16.278 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.memtable_factory: SkipListFactory 2023-07-14T18:57:16.278 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.table_factory: BlockBasedTable 2023-07-14T18:57:16.278 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5567c25f73a0) 2023-07-14T18:57:16.278 INFO:journalctl@ceph.mon.b.smithi088.stdout: cache_index_and_filter_blocks: 1 2023-07-14T18:57:16.278 INFO:journalctl@ceph.mon.b.smithi088.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-07-14T18:57:16.278 INFO:journalctl@ceph.mon.b.smithi088.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-07-14T18:57:16.278 INFO:journalctl@ceph.mon.b.smithi088.stdout: pin_top_level_index_and_filter: 1 2023-07-14T18:57:16.279 INFO:journalctl@ceph.mon.b.smithi088.stdout: index_type: 0 2023-07-14T18:57:16.279 INFO:journalctl@ceph.mon.b.smithi088.stdout: data_block_index_type: 0 2023-07-14T18:57:16.279 INFO:journalctl@ceph.mon.b.smithi088.stdout: index_shortening: 1 2023-07-14T18:57:16.279 INFO:journalctl@ceph.mon.b.smithi088.stdout: data_block_hash_table_util_ratio: 0.750000 2023-07-14T18:57:16.279 INFO:journalctl@ceph.mon.b.smithi088.stdout: checksum: 4 2023-07-14T18:57:16.279 INFO:journalctl@ceph.mon.b.smithi088.stdout: no_block_cache: 0 2023-07-14T18:57:16.279 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_cache: 0x5567c25b71f0 2023-07-14T18:57:16.279 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_cache_name: BinnedLRUCache 2023-07-14T18:57:16.280 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_cache_options: 2023-07-14T18:57:16.280 INFO:journalctl@ceph.mon.b.smithi088.stdout: capacity : 536870912 2023-07-14T18:57:16.280 INFO:journalctl@ceph.mon.b.smithi088.stdout: num_shard_bits : 4 2023-07-14T18:57:16.280 INFO:journalctl@ceph.mon.b.smithi088.stdout: strict_capacity_limit : 0 2023-07-14T18:57:16.280 INFO:journalctl@ceph.mon.b.smithi088.stdout: high_pri_pool_ratio: 0.000 2023-07-14T18:57:16.280 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_cache_compressed: (nil) 2023-07-14T18:57:16.280 INFO:journalctl@ceph.mon.b.smithi088.stdout: persistent_cache: (nil) 2023-07-14T18:57:16.281 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_size: 4096 2023-07-14T18:57:16.281 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_size_deviation: 10 2023-07-14T18:57:16.281 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_restart_interval: 16 2023-07-14T18:57:16.281 INFO:journalctl@ceph.mon.b.smithi088.stdout: index_block_restart_interval: 1 2023-07-14T18:57:16.281 INFO:journalctl@ceph.mon.b.smithi088.stdout: metadata_block_size: 4096 2023-07-14T18:57:16.281 INFO:journalctl@ceph.mon.b.smithi088.stdout: partition_filters: 0 2023-07-14T18:57:16.281 INFO:journalctl@ceph.mon.b.smithi088.stdout: use_delta_encoding: 1 2023-07-14T18:57:16.281 INFO:journalctl@ceph.mon.b.smithi088.stdout: filter_policy: bloomfilter 2023-07-14T18:57:16.282 INFO:journalctl@ceph.mon.b.smithi088.stdout: whole_key_filtering: 1 2023-07-14T18:57:16.282 INFO:journalctl@ceph.mon.b.smithi088.stdout: verify_compression: 0 2023-07-14T18:57:16.282 INFO:journalctl@ceph.mon.b.smithi088.stdout: read_amp_bytes_per_bit: 0 2023-07-14T18:57:16.282 INFO:journalctl@ceph.mon.b.smithi088.stdout: format_version: 5 2023-07-14T18:57:16.282 INFO:journalctl@ceph.mon.b.smithi088.stdout: enable_index_compression: 1 2023-07-14T18:57:16.282 INFO:journalctl@ceph.mon.b.smithi088.stdout: block_align: 0 2023-07-14T18:57:16.282 INFO:journalctl@ceph.mon.b.smithi088.stdout: max_auto_readahead_size: 262144 2023-07-14T18:57:16.283 INFO:journalctl@ceph.mon.b.smithi088.stdout: prepopulate_block_cache: 0 2023-07-14T18:57:16.283 INFO:journalctl@ceph.mon.b.smithi088.stdout: initial_auto_readahead_size: 8192 2023-07-14T18:57:16.283 INFO:journalctl@ceph.mon.b.smithi088.stdout: num_file_reads_for_auto_readahead: 2 2023-07-14T18:57:16.283 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.write_buffer_size: 33554432 2023-07-14T18:57:16.283 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_write_buffer_number: 2 2023-07-14T18:57:16.283 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression: NoCompression 2023-07-14T18:57:16.283 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression: Disabled 2023-07-14T18:57:16.283 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.prefix_extractor: nullptr 2023-07-14T18:57:16.284 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-07-14T18:57:16.284 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.num_levels: 7 2023-07-14T18:57:16.284 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-07-14T18:57:16.284 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-07-14T18:57:16.284 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-07-14T18:57:16.284 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-07-14T18:57:16.284 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-07-14T18:57:16.284 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-07-14T18:57:16.285 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-07-14T18:57:16.285 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:57:16.285 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-07-14T18:57:16.285 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-07-14T18:57:16.285 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:57:16.285 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:57:16.285 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.window_bits: -14 2023-07-14T18:57:16.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.level: 32767 2023-07-14T18:57:16.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.strategy: 0 2023-07-14T18:57:16.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-07-14T18:57:16.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-07-14T18:57:16.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-07-14T18:57:16.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-07-14T18:57:16.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.enabled: false 2023-07-14T18:57:16.286 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-07-14T18:57:16.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-07-14T18:57:16.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-07-14T18:57:16.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-07-14T18:57:16.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.target_file_size_base: 67108864 2023-07-14T18:57:16.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.target_file_size_multiplier: 1 2023-07-14T18:57:16.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-07-14T18:57:16.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-07-14T18:57:16.287 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-07-14T18:57:16.288 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-07-14T18:57:16.288 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-07-14T18:57:16.288 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-07-14T18:57:16.288 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-07-14T18:57:16.288 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-07-14T18:57:16.288 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-07-14T18:57:16.288 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-07-14T18:57:16.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-07-14T18:57:16.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-07-14T18:57:16.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-07-14T18:57:16.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.arena_block_size: 1048576 2023-07-14T18:57:16.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-07-14T18:57:16.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-07-14T18:57:16.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.disable_auto_compactions: 0 2023-07-14T18:57:16.289 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-07-14T18:57:16.290 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-07-14T18:57:16.290 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-07-14T18:57:16.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-07-14T18:57:16.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-07-14T18:57:16.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-07-14T18:57:16.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-07-14T18:57:16.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-07-14T18:57:16.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-07-14T18:57:16.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-07-14T18:57:16.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.table_properties_collectors: 2023-07-14T18:57:16.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.inplace_update_support: 0 2023-07-14T18:57:16.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.inplace_update_num_locks: 10000 2023-07-14T18:57:16.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-07-14T18:57:16.292 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-07-14T18:57:16.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.memtable_huge_page_size: 0 2023-07-14T18:57:16.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.bloom_locality: 0 2023-07-14T18:57:16.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.max_successive_merges: 0 2023-07-14T18:57:16.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.optimize_filters_for_hits: 0 2023-07-14T18:57:16.293 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.paranoid_file_checks: 0 2023-07-14T18:57:16.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.force_consistency_checks: 1 2023-07-14T18:57:16.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.report_bg_io_stats: 0 2023-07-14T18:57:16.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.ttl: 2592000 2023-07-14T18:57:16.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.periodic_compaction_seconds: 0 2023-07-14T18:57:16.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-07-14T18:57:16.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-07-14T18:57:16.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.enable_blob_files: false 2023-07-14T18:57:16.294 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.min_blob_size: 0 2023-07-14T18:57:16.295 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.blob_file_size: 268435456 2023-07-14T18:57:16.295 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.blob_compression_type: NoCompression 2023-07-14T18:57:16.295 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.enable_blob_garbage_collection: false 2023-07-14T18:57:16.295 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-07-14T18:57:16.295 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-07-14T18:57:16.295 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-07-14T18:57:16.295 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.blob_file_starting_level: 0 2023-07-14T18:57:16.296 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-07-14T18:57:16.296 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-07-14T18:57:16.296 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-07-14T18:57:16.296 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 0cabf90a-d1b6-4535-8e1f-a517525b6495 2023-07-14T18:57:16.296 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361036193691, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-07-14T18:57:16.296 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-07-14T18:57:16.296 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361036194095, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1689361036, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "0cabf90a-d1b6-4535-8e1f-a517525b6495", "db_session_id": "TXIRROTI1KTGNBOO2PQQ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-07-14T18:57:16.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: EVENT_LOG_v1 {"time_micros": 1689361036194159, "job": 1, "event": "recovery_finished"} 2023-07-14T18:57:16.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-07-14T18:57:16.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-07-14T18:57:16.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5567c2650000 2023-07-14T18:57:16.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: DB pointer 0x5567c263c000 2023-07-14T18:57:16.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-07-14T18:57:16.297 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-07-14T18:57:16.297 INFO:journalctl@ceph.mon.b.smithi088.stdout: ** DB Stats ** 2023-07-14T18:57:16.298 INFO:journalctl@ceph.mon.b.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:57:16.298 INFO:journalctl@ceph.mon.b.smithi088.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-07-14T18:57:16.298 INFO:journalctl@ceph.mon.b.smithi088.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:57:16.298 INFO:journalctl@ceph.mon.b.smithi088.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:57:16.298 INFO:journalctl@ceph.mon.b.smithi088.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-07-14T18:57:16.298 INFO:journalctl@ceph.mon.b.smithi088.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-07-14T18:57:16.298 INFO:journalctl@ceph.mon.b.smithi088.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-07-14T18:57:16.298 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-14T18:57:16.299 INFO:journalctl@ceph.mon.b.smithi088.stdout: ** Compaction Stats [default] ** 2023-07-14T18:57:16.299 INFO:journalctl@ceph.mon.b.smithi088.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:57:16.299 INFO:journalctl@ceph.mon.b.smithi088.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-07-14T18:57:16.299 INFO:journalctl@ceph.mon.b.smithi088.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:57:16.299 INFO:journalctl@ceph.mon.b.smithi088.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:57:16.300 INFO:journalctl@ceph.mon.b.smithi088.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:57:16.300 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-14T18:57:16.300 INFO:journalctl@ceph.mon.b.smithi088.stdout: ** Compaction Stats [default] ** 2023-07-14T18:57:16.300 INFO:journalctl@ceph.mon.b.smithi088.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-07-14T18:57:16.300 INFO:journalctl@ceph.mon.b.smithi088.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-07-14T18:57:16.300 INFO:journalctl@ceph.mon.b.smithi088.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.3 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-07-14T18:57:16.300 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-14T18:57:16.300 INFO:journalctl@ceph.mon.b.smithi088.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-07-14T18:57:16.301 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-14T18:57:16.301 INFO:journalctl@ceph.mon.b.smithi088.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-07-14T18:57:16.301 INFO:journalctl@ceph.mon.b.smithi088.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-07-14T18:57:16.301 INFO:journalctl@ceph.mon.b.smithi088.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-07-14T18:57:16.301 INFO:journalctl@ceph.mon.b.smithi088.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-07-14T18:57:16.301 INFO:journalctl@ceph.mon.b.smithi088.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-07-14T18:57:16.301 INFO:journalctl@ceph.mon.b.smithi088.stdout: AddFile(Keys): cumulative 0, interval 0 2023-07-14T18:57:16.302 INFO:journalctl@ceph.mon.b.smithi088.stdout: Cumulative compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:57:16.302 INFO:journalctl@ceph.mon.b.smithi088.stdout: Interval compaction: 0.00 GB write, 0.69 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-07-14T18:57:16.302 INFO:journalctl@ceph.mon.b.smithi088.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-07-14T18:57:16.302 INFO:journalctl@ceph.mon.b.smithi088.stdout: Block cache BinnedLRUCache@0x5567c25b71f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2023-07-14T18:57:16.302 INFO:journalctl@ceph.mon.b.smithi088.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-07-14T18:57:16.302 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-14T18:57:16.302 INFO:journalctl@ceph.mon.b.smithi088.stdout: ** File Read Latency Histogram By Level [default] ** 2023-07-14T18:57:16.302 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-07-14T18:57:16.303 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: using public_addr v2:172.21.15.88:0/0 -> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] 2023-07-14T18:57:16.303 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: starting mon.b rank -1 at public addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] at bind addrs [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:57:16.303 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(???) e0 preinit fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:57:16.303 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).mds e1 new map 2023-07-14T18:57:16.303 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).mds e1 print_map 2023-07-14T18:57:16.303 INFO:journalctl@ceph.mon.b.smithi088.stdout: e1 2023-07-14T18:57:16.303 INFO:journalctl@ceph.mon.b.smithi088.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T18:57:16.304 INFO:journalctl@ceph.mon.b.smithi088.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T18:57:16.304 INFO:journalctl@ceph.mon.b.smithi088.stdout: legacy client fscid: -1 2023-07-14T18:57:16.304 INFO:journalctl@ceph.mon.b.smithi088.stdout: 2023-07-14T18:57:16.304 INFO:journalctl@ceph.mon.b.smithi088.stdout: No filesystems configured 2023-07-14T18:57:16.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-07-14T18:57:16.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-07-14T18:57:16.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-07-14T18:57:16.304 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-07-14T18:57:16.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-07-14T18:57:16.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-07-14T18:57:16.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-07-14T18:57:16.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:57:16.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:57:16.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-07-14T18:57:16.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.305 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:16.306 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.306 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.306 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.306 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.306 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi088", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:57:16.306 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:16.306 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:16.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T18:57:16.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:16.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Updating smithi088:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:57:16.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Deploying cephadm binary to smithi102 2023-07-14T18:57:16.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:57:16.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Added host smithi102 2023-07-14T18:57:16.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:16.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T18:57:16.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.309 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:57:16.310 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:16.310 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:16.310 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1181204817' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-07-14T18:57:16.310 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Updating smithi102:/etc/ceph/ceph.conf 2023-07-14T18:57:16.310 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:16.310 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:57:16.310 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.311 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.311 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.311 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1181204817' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-07-14T18:57:16.311 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:16.311 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.client.admin.keyring 2023-07-14T18:57:16.311 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.311 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.312 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:16.312 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Saving service mon spec with placement smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c;count:3 2023-07-14T18:57:16.312 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.312 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:16.312 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:16.312 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:16.312 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.312 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:16.313 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:16.313 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Deploying daemon mon.c on smithi102 2023-07-14T18:57:16.313 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.313 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/62612413' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:57:16.313 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:57:16.313 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.a calling monitor election 2023-07-14T18:57:16.313 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:16.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:16.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:16.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.c calling monitor election 2023-07-14T18:57:16.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:16.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:16.314 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:16.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:57:16.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: monmap e2: 2 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:16.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: fsmap 2023-07-14T18:57:16.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:16.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mgrmap e13: x(active, since 42s) 2023-07-14T18:57:16.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: overall HEALTH_OK 2023-07-14T18:57:16.315 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.316 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.316 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:16.316 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:16.316 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:16.316 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: Deploying daemon mon.b on smithi088 2023-07-14T18:57:16.316 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/607165267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-07-14T18:57:16.316 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:16.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:16.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-07-14T18:57:16.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-07-14T18:57:16.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta expand map: {default=false} 2023-07-14T18:57:16.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta from 'false' to 'false' 2023-07-14T18:57:16.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta expanded map: {default=false} 2023-07-14T18:57:16.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta expand map: {default=info} 2023-07-14T18:57:16.317 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta from 'info' to 'info' 2023-07-14T18:57:16.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta expanded map: {default=info} 2023-07-14T18:57:16.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta expand map: {default=daemon} 2023-07-14T18:57:16.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta from 'daemon' to 'daemon' 2023-07-14T18:57:16.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta expanded map: {default=daemon} 2023-07-14T18:57:16.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta expand map: {default=debug} 2023-07-14T18:57:16.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta from 'debug' to 'debug' 2023-07-14T18:57:16.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 ceph-mon[128973]: expand_channel_meta expanded map: {default=debug} 2023-07-14T18:57:16.318 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:16 smithi088 systemd[1]: Started Ceph mon.b for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T18:57:21.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:21.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:57:21.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: mon.c calling monitor election 2023-07-14T18:57:21.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: mon.a calling monitor election 2023-07-14T18:57:21.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:21.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:21.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:21.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:57:21.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:21.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: fsmap 2023-07-14T18:57:21.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:21.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: mgrmap e13: x(active, since 51s) 2023-07-14T18:57:21.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T18:57:21.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T18:57:21.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T18:57:21.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: mon.b (rank 2) addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] is down (out of quorum) 2023-07-14T18:57:21.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:21.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:21.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:21.636 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:57:21.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: mon.c calling monitor election 2023-07-14T18:57:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: mon.a calling monitor election 2023-07-14T18:57:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:57:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: fsmap 2023-07-14T18:57:21.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: mgrmap e13: x(active, since 51s) 2023-07-14T18:57:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T18:57:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T18:57:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T18:57:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: mon.b (rank 2) addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] is down (out of quorum) 2023-07-14T18:57:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:21.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:21.803 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:22.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:22 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:22.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:22 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:22.659 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.654+0000 7f6df692a700 1 -- 172.21.15.102:0/18375660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd8008440 msgr2=0x7f6dd800a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.654+0000 7f6df692a700 1 --2- 172.21.15.102:0/18375660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd8008440 0x7f6dd800a8a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6df00ff800 tx=0x7f6de400fe20 comp rx=0 tx=0).stop 2023-07-14T18:57:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.654+0000 7f6df692a700 1 -- 172.21.15.102:0/18375660 shutdown_connections 2023-07-14T18:57:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.654+0000 7f6df692a700 1 --2- 172.21.15.102:0/18375660 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6dd8008440 0x7f6dd800a8a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.654+0000 7f6df692a700 1 -- 172.21.15.102:0/18375660 >> 172.21.15.102:0/18375660 conn(0x7f6df00fa260 msgr2=0x7f6df00fc670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:22.660 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.654+0000 7f6df692a700 1 -- 172.21.15.102:0/18375660 shutdown_connections 2023-07-14T18:57:22.661 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.654+0000 7f6df692a700 1 -- 172.21.15.102:0/18375660 wait complete. 2023-07-14T18:57:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df692a700 1 Processor -- start 2023-07-14T18:57:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df692a700 1 -- start start 2023-07-14T18:57:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df692a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 0x7f6df0196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df692a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 0x7f6df019b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:22.663 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df692a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df019be60 0x7f6df019c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df692a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df0197b80 con 0x7f6df019be60 2023-07-14T18:57:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df692a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df01a3bd0 con 0x7f6df01974b0 2023-07-14T18:57:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df692a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6df01a3d10 con 0x7f6dd8008440 2023-07-14T18:57:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6deffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 0x7f6df0196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6deffff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 0x7f6df0196f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52014/0 (socket says 172.21.15.102:52014) 2023-07-14T18:57:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6deffff700 1 -- 172.21.15.102:0/3734991750 learned_addr learned my addr 172.21.15.102:0/3734991750 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T18:57:22.664 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6deffff700 1 -- 172.21.15.102:0/3734991750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 msgr2=0x7f6df0196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2023-07-14T18:57:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6deffff700 1 -- 172.21.15.102:0/3734991750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 msgr2=0x7f6df0196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:57:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6deffff700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 0x7f6df0196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:57:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6deffff700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 0x7f6df0196f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:57:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6def7fe700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 0x7f6df019b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.655+0000 7f6df4ec7700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df019be60 0x7f6df019c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6def7fe700 1 -- 172.21.15.102:0/3734991750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 msgr2=0x7f6df019b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2023-07-14T18:57:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6def7fe700 1 -- 172.21.15.102:0/3734991750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 msgr2=0x7f6df019b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-07-14T18:57:22.665 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6def7fe700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 0x7f6df019b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-07-14T18:57:22.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6def7fe700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 0x7f6df019b920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-07-14T18:57:22.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6df4ec7700 1 -- 172.21.15.102:0/3734991750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 msgr2=0x7f6df0196f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:57:22.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6df4ec7700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 0x7f6df0196f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:22.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6df4ec7700 1 -- 172.21.15.102:0/3734991750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 msgr2=0x7f6df019b920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-07-14T18:57:22.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6df4ec7700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 0x7f6df019b920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:22.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6df4ec7700 1 -- 172.21.15.102:0/3734991750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6de400f870 con 0x7f6df019be60 2023-07-14T18:57:22.666 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6df4ec7700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df019be60 0x7f6df019c300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6de800bd80 tx=0x7f6de800be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6ded7fa700 1 -- 172.21.15.102:0/3734991750 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de800cd10 con 0x7f6df019be60 2023-07-14T18:57:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6df01a3fc0 con 0x7f6df019be60 2023-07-14T18:57:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6ded7fa700 1 -- 172.21.15.102:0/3734991750 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6de800ce70 con 0x7f6df019be60 2023-07-14T18:57:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6ded7fa700 1 -- 172.21.15.102:0/3734991750 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6de8012700 con 0x7f6df019be60 2023-07-14T18:57:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.656+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6df01a4460 con 0x7f6df019be60 2023-07-14T18:57:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.657+0000 7f6ded7fa700 1 -- 172.21.15.102:0/3734991750 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f6de80128e0 con 0x7f6df019be60 2023-07-14T18:57:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.657+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6df0061c70 con 0x7f6df019be60 2023-07-14T18:57:22.667 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.657+0000 7f6ded7fa700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ddc038c30 0x7f6ddc03b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:22.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.657+0000 7f6ded7fa700 1 -- 172.21.15.102:0/3734991750 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f6de804ad20 con 0x7f6df019be60 2023-07-14T18:57:22.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.658+0000 7f6deffff700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ddc038c30 0x7f6ddc03b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:22.668 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.658+0000 7f6deffff700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ddc038c30 0x7f6ddc03b0e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6de4001eb0 tx=0x7f6de4005c40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:22.669 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.663+0000 7f6ded7fa700 1 -- 172.21.15.102:0/3734991750 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6de801d3b0 con 0x7f6df019be60 2023-07-14T18:57:22.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:22 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:22.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:22 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:22.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.936+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6df0069350 con 0x7f6df019be60 2023-07-14T18:57:22.942 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.937+0000 7f6ded7fa700 1 -- 172.21.15.102:0/3734991750 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1270 (secure 0 0 0) 0x7f6de80112a0 con 0x7f6df019be60 2023-07-14T18:57:22.942 INFO:teuthology.orchestra.run.smithi102.stdout: 2023-07-14T18:57:22.943 INFO:teuthology.orchestra.run.smithi102.stdout:{"epoch":3,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","modified":"2023-07-14T18:57:16.219317Z","created":"2023-07-14T18:55:32.052335Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:3300","nonce":0},{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:3300","nonce":0},{"type":"v1","addr":"172.21.15.88:6789","nonce":0}]},"addr":"172.21.15.88:6789/0","public_addr":"172.21.15.88:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-07-14T18:57:22.944 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ddc038c30 msgr2=0x7f6ddc03b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ddc038c30 0x7f6ddc03b0e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6de4001eb0 tx=0x7f6de4005c40 comp rx=0 tx=0).stop 2023-07-14T18:57:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df019be60 msgr2=0x7f6df019c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df019be60 0x7f6df019c300 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6de800bd80 tx=0x7f6de800be60 comp rx=0 tx=0).stop 2023-07-14T18:57:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 shutdown_connections 2023-07-14T18:57:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ddc038c30 0x7f6ddc03b0e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:22.945 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6df019be60 0x7f6df019c300 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6dd8008440 0x7f6df0196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 --2- 172.21.15.102:0/3734991750 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6df01974b0 0x7f6df019b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 >> 172.21.15.102:0/3734991750 conn(0x7f6df00fa260 msgr2=0x7f6df0100380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 shutdown_connections 2023-07-14T18:57:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:22.939+0000 7f6df692a700 1 -- 172.21.15.102:0/3734991750 wait complete. 2023-07-14T18:57:22.946 INFO:teuthology.orchestra.run.smithi102.stderr:dumped monmap epoch 3 2023-07-14T18:57:23.366 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-07-14T18:57:23.366 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph config generate-minimal-conf 2023-07-14T18:57:23.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:23.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:57:23.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.c calling monitor election 2023-07-14T18:57:23.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.a calling monitor election 2023-07-14T18:57:23.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:23.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:57:23.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:23.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:23.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:23.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:23.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:23.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:23.716 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:23.716 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-07-14T18:57:23.716 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:23.716 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: fsmap 2023-07-14T18:57:23.716 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:23.716 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mgrmap e13: x(active, since 51s) 2023-07-14T18:57:23.716 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2023-07-14T18:57:23.717 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2023-07-14T18:57:23.717 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2023-07-14T18:57:23.717 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.b (rank 2) addr [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] is down (out of quorum) 2023-07-14T18:57:23.717 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:23.717 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:23.717 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:23.717 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:23.718 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:23.718 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:23.993 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:57:24.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.b calling monitor election 2023-07-14T18:57:24.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Updating smithi078:/etc/ceph/ceph.conf 2023-07-14T18:57:24.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T18:57:24.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Updating smithi102:/etc/ceph/ceph.conf 2023-07-14T18:57:24.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Updating smithi078:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.a calling monitor election 2023-07-14T18:57:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.c calling monitor election 2023-07-14T18:57:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.b calling monitor election 2023-07-14T18:57:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:57:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: fsmap 2023-07-14T18:57:24.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:24.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: mgrmap e13: x(active, since 53s) 2023-07-14T18:57:24.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T18:57:24.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: Cluster is now healthy 2023-07-14T18:57:24.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: overall HEALTH_OK 2023-07-14T18:57:24.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.216 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.217 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.217 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.217 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.217 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.217 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.217 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:24.217 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:24.218 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:24.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: mon.b calling monitor election 2023-07-14T18:57:24.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: Updating smithi078:/etc/ceph/ceph.conf 2023-07-14T18:57:24.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T18:57:24.256 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: Updating smithi102:/etc/ceph/ceph.conf 2023-07-14T18:57:24.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: Updating smithi078:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: mon.a calling monitor election 2023-07-14T18:57:24.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: mon.c calling monitor election 2023-07-14T18:57:24.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: mon.b calling monitor election 2023-07-14T18:57:24.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:57:24.257 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:24.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: fsmap 2023-07-14T18:57:24.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:24.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: mgrmap e13: x(active, since 53s) 2023-07-14T18:57:24.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T18:57:24.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: Cluster is now healthy 2023-07-14T18:57:24.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: overall HEALTH_OK 2023-07-14T18:57:24.258 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.259 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:24.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:24.260 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:24.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: mon.b calling monitor election 2023-07-14T18:57:24.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: Updating smithi078:/etc/ceph/ceph.conf 2023-07-14T18:57:24.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: Updating smithi088:/etc/ceph/ceph.conf 2023-07-14T18:57:24.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: Updating smithi102:/etc/ceph/ceph.conf 2023-07-14T18:57:24.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: Updating smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: Updating smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: Updating smithi078:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/config/ceph.conf 2023-07-14T18:57:24.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: mon.a calling monitor election 2023-07-14T18:57:24.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: mon.c calling monitor election 2023-07-14T18:57:24.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: mon.b calling monitor election 2023-07-14T18:57:24.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-07-14T18:57:24.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: monmap e3: 3 mons at {a=[v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0],b=[v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0],c=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0]} removed_ranks: {} 2023-07-14T18:57:24.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: fsmap 2023-07-14T18:57:24.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: osdmap e4: 0 total, 0 up, 0 in 2023-07-14T18:57:24.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: mgrmap e13: x(active, since 53s) 2023-07-14T18:57:24.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2023-07-14T18:57:24.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: Cluster is now healthy 2023-07-14T18:57:24.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: overall HEALTH_OK 2023-07-14T18:57:24.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:24.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:24.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:24.386 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:25.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:57:25.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: Reconfiguring daemon mon.a on smithi078 2023-07-14T18:57:25.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:25.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:25.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:25.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:25.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:25.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:25.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:57:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: Reconfiguring daemon mon.a on smithi078 2023-07-14T18:57:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:25.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: Reconfiguring mon.a (unknown last config time)... 2023-07-14T18:57:25.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: Reconfiguring daemon mon.a on smithi078 2023-07-14T18:57:25.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:25.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:57:25.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:25.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:25.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:25.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:25.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:25.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 -- 172.21.15.78:0/2803710104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 msgr2=0x7fcb4c10ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:25.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 --2- 172.21.15.78:0/2803710104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 0x7fcb4c10ba60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fcb4400b0d0 tx=0x7fcb4400b3e0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 -- 172.21.15.78:0/2803710104 shutdown_connections 2023-07-14T18:57:25.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 --2- 172.21.15.78:0/2803710104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 0x7fcb4c10ba60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 --2- 172.21.15.78:0/2803710104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0fee40 0x7fcb4c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.640 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 --2- 172.21.15.78:0/2803710104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0ff7e0 0x7fcb4c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 -- 172.21.15.78:0/2803710104 >> 172.21.15.78:0/2803710104 conn(0x7fcb4c075ca0 msgr2=0x7fcb4c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:25.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 -- 172.21.15.78:0/2803710104 shutdown_connections 2023-07-14T18:57:25.641 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.634+0000 7fcb53e94700 1 -- 172.21.15.78:0/2803710104 wait complete. 2023-07-14T18:57:25.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb53e94700 1 Processor -- start 2023-07-14T18:57:25.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb53e94700 1 -- start start 2023-07-14T18:57:25.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb53e94700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0fee40 0x7fcb4c196c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:25.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb53e94700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0ff7e0 0x7fcb4c1971a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:25.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb53e94700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 0x7fcb4c1a1ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:25.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb53e94700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c197ab0 con 0x7fcb4c103ee0 2023-07-14T18:57:25.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb53e94700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1a2220 con 0x7fcb4c0ff7e0 2023-07-14T18:57:25.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb53e94700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c1a2390 con 0x7fcb4c0fee40 2023-07-14T18:57:25.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb52431700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 0x7fcb4c1a1ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:25.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb52431700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 0x7fcb4c1a1ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33562/0 (socket says 172.21.15.78:33562) 2023-07-14T18:57:25.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb52431700 1 -- 172.21.15.78:0/397612754 learned_addr learned my addr 172.21.15.78:0/397612754 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:57:25.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb5142f700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0ff7e0 0x7fcb4c1971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:25.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb51c30700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0fee40 0x7fcb4c196c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:25.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb52431700 1 -- 172.21.15.78:0/397612754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0fee40 msgr2=0x7fcb4c196c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:25.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb52431700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0fee40 0x7fcb4c196c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb52431700 1 -- 172.21.15.78:0/397612754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0ff7e0 msgr2=0x7fcb4c1971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:25.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb52431700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0ff7e0 0x7fcb4c1971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.635+0000 7fcb52431700 1 -- 172.21.15.78:0/397612754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb44009d00 con 0x7fcb4c103ee0 2023-07-14T18:57:25.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.636+0000 7fcb52431700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 0x7fcb4c1a1ce0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009330 tx=0x7fcb44009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:25.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.636+0000 7fcb42ffd700 1 -- 172.21.15.78:0/397612754 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4400e040 con 0x7fcb4c103ee0 2023-07-14T18:57:25.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.636+0000 7fcb42ffd700 1 -- 172.21.15.78:0/397612754 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcb44003e10 con 0x7fcb4c103ee0 2023-07-14T18:57:25.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.636+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c1a2610 con 0x7fcb4c103ee0 2023-07-14T18:57:25.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.636+0000 7fcb42ffd700 1 -- 172.21.15.78:0/397612754 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4401a7f0 con 0x7fcb4c103ee0 2023-07-14T18:57:25.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.636+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c1a2a20 con 0x7fcb4c103ee0 2023-07-14T18:57:25.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.637+0000 7fcb42ffd700 1 -- 172.21.15.78:0/397612754 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7fcb44005070 con 0x7fcb4c103ee0 2023-07-14T18:57:25.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.637+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c04e340 con 0x7fcb4c103ee0 2023-07-14T18:57:25.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.637+0000 7fcb42ffd700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcb380389e0 0x7fcb3803ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:25.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.637+0000 7fcb42ffd700 1 -- 172.21.15.78:0/397612754 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fcb4404c5e0 con 0x7fcb4c103ee0 2023-07-14T18:57:25.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.637+0000 7fcb51c30700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcb380389e0 0x7fcb3803ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:25.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.637+0000 7fcb51c30700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcb380389e0 0x7fcb3803ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c005950 tx=0x7fcb3c005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:25.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.641+0000 7fcb42ffd700 1 -- 172.21.15.78:0/397612754 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcb4402ad30 con 0x7fcb4c103ee0 2023-07-14T18:57:25.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.825+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fcb4c1a2d00 con 0x7fcb4c103ee0 2023-07-14T18:57:25.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.826+0000 7fcb42ffd700 1 -- 172.21.15.78:0/397612754 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+273 (secure 0 0 0) 0x7fcb4401e020 con 0x7fcb4c103ee0 2023-07-14T18:57:25.831 INFO:teuthology.orchestra.run.smithi078.stdout:# minimal ceph.conf for e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:57:25.832 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2023-07-14T18:57:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T18:57:25.832 INFO:teuthology.orchestra.run.smithi078.stdout: mon_host = [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] 2023-07-14T18:57:25.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcb380389e0 msgr2=0x7fcb3803ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:25.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcb380389e0 0x7fcb3803ae90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcb3c005950 tx=0x7fcb3c005cf0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 msgr2=0x7fcb4c1a1ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:25.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 0x7fcb4c1a1ce0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009330 tx=0x7fcb44009360 comp rx=0 tx=0).stop 2023-07-14T18:57:25.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 shutdown_connections 2023-07-14T18:57:25.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcb380389e0 0x7fcb3803ae90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcb4c103ee0 0x7fcb4c1a1ce0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb4c0fee40 0x7fcb4c196c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 --2- 172.21.15.78:0/397612754 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcb4c0ff7e0 0x7fcb4c1971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:25.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 >> 172.21.15.78:0/397612754 conn(0x7fcb4c075ca0 msgr2=0x7fcb4c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:25.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 shutdown_connections 2023-07-14T18:57:25.835 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:25.828+0000 7fcb53e94700 1 -- 172.21.15.78:0/397612754 wait complete. 2023-07-14T18:57:26.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:57:26.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: Reconfiguring daemon mon.b on smithi088 2023-07-14T18:57:26.230 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/397612754' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:26.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:26.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:26.231 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:26.285 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-07-14T18:57:26.286 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:57:26.286 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:57:26.315 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:57:26.316 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:57:26.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:57:26.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: Reconfiguring daemon mon.b on smithi088 2023-07-14T18:57:26.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/397612754' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:26.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:26.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:26.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:26.384 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:57:26.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:57:26.411 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: Reconfiguring mon.b (monmap changed)... 2023-07-14T18:57:26.411 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: Reconfiguring daemon mon.b on smithi088 2023-07-14T18:57:26.412 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.412 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/397612754' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:26.412 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.412 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:26.412 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-07-14T18:57:26.412 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-07-14T18:57:26.412 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:26.415 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:57:26.415 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:57:26.482 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:57:26.482 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2023-07-14T18:57:26.512 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:57:26.513 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-07-14T18:57:26.579 INFO:tasks.cephadm:Adding mgr.x on smithi078 2023-07-14T18:57:26.580 INFO:tasks.cephadm:Adding mgr.y on smithi088 2023-07-14T18:57:26.580 INFO:tasks.cephadm:Adding mgr.z on smithi102 2023-07-14T18:57:26.581 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch apply mgr '3;smithi078=x;smithi088=y;smithi102=z' 2023-07-14T18:57:27.311 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:57:27.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:27 smithi102 ceph-mon[129280]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:57:27.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:27 smithi102 ceph-mon[129280]: Reconfiguring daemon mon.c on smithi102 2023-07-14T18:57:27.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:27 smithi102 ceph-mon[129280]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:27.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:27 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:27.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:27 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:27 smithi088 ceph-mon[128973]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:57:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:27 smithi088 ceph-mon[128973]: Reconfiguring daemon mon.c on smithi102 2023-07-14T18:57:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:27 smithi088 ceph-mon[128973]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:27.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:27 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:27.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:27 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:27.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:27 smithi078 ceph-mon[127425]: Reconfiguring mon.c (monmap changed)... 2023-07-14T18:57:27.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:27 smithi078 ceph-mon[127425]: Reconfiguring daemon mon.c on smithi102 2023-07-14T18:57:27.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:27 smithi078 ceph-mon[127425]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:27.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:27 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:27.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:27 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:28.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:28.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:28.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:28.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:28.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:28.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:28.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:28.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:28.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 -- 172.21.15.102:0/1950981393 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 msgr2=0x7f77e4107000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:28.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/1950981393 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 0x7f77e4107000 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f77dc00b0a0 tx=0x7f77dc00b3b0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.505 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 -- 172.21.15.102:0/1950981393 shutdown_connections 2023-07-14T18:57:28.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/1950981393 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77e40febb0 0x7f77e40fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/1950981393 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 0x7f77e4107000 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/1950981393 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77e40ffdb0 0x7f77e4100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 -- 172.21.15.102:0/1950981393 >> 172.21.15.102:0/1950981393 conn(0x7f77e40fa1e0 msgr2=0x7f77e40fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:28.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 -- 172.21.15.102:0/1950981393 shutdown_connections 2023-07-14T18:57:28.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.498+0000 7f77eb6d7700 1 -- 172.21.15.102:0/1950981393 wait complete. 2023-07-14T18:57:28.506 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77eb6d7700 1 Processor -- start 2023-07-14T18:57:28.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77eb6d7700 1 -- start start 2023-07-14T18:57:28.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77eb6d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77e40febb0 0x7f77e4069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:28.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77eb6d7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77e40ffdb0 0x7f77e419a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:28.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77eb6d7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 0x7f77e419cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:28.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77eb6d7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e419d530 con 0x7f77e40febb0 2023-07-14T18:57:28.507 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77eb6d7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e419d6a0 con 0x7f77e40ffdb0 2023-07-14T18:57:28.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77eb6d7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e419d860 con 0x7f77e4100740 2023-07-14T18:57:28.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 0x7f77e419cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:28.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 0x7f77e419cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52030/0 (socket says 172.21.15.102:52030) 2023-07-14T18:57:28.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 -- 172.21.15.102:0/3790399890 learned_addr learned my addr 172.21.15.102:0/3790399890 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T18:57:28.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9473700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77e40febb0 0x7f77e4069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:28.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e8c72700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77e40ffdb0 0x7f77e419a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:28.508 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 -- 172.21.15.102:0/3790399890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77e40ffdb0 msgr2=0x7f77e419a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:28.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77e40ffdb0 0x7f77e419a700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 -- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77e40febb0 msgr2=0x7f77e4069550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:28.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77e40febb0 0x7f77e4069550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 -- 172.21.15.102:0/3790399890 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77dc009d00 con 0x7f77e4100740 2023-07-14T18:57:28.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.499+0000 7f77e9c74700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 0x7f77e419cd90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0062a0 tx=0x7f77dc004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:28.509 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.500+0000 7f77d67fc700 1 -- 172.21.15.102:0/3790399890 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc00e040 con 0x7f77e4100740 2023-07-14T18:57:28.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.500+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77e419dae0 con 0x7f77e4100740 2023-07-14T18:57:28.510 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.500+0000 7f77d67fc700 1 -- 172.21.15.102:0/3790399890 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f77dc003e90 con 0x7f77e4100740 2023-07-14T18:57:28.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.500+0000 7f77d67fc700 1 -- 172.21.15.102:0/3790399890 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77dc004020 con 0x7f77e4100740 2023-07-14T18:57:28.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.500+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e419df50 con 0x7f77e4100740 2023-07-14T18:57:28.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.501+0000 7f77d67fc700 1 -- 172.21.15.102:0/3790399890 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f77dc003a00 con 0x7f77e4100740 2023-07-14T18:57:28.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.501+0000 7f77d67fc700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77d00389e0 0x7f77d003ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:28.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.501+0000 7f77d67fc700 1 -- 172.21.15.102:0/3790399890 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f77dc04cf50 con 0x7f77e4100740 2023-07-14T18:57:28.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.501+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e410a230 con 0x7f77e4100740 2023-07-14T18:57:28.511 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.501+0000 7f77e9473700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77d00389e0 0x7f77d003ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:28.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.502+0000 7f77e9473700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77d00389e0 0x7f77d003ae90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f77e40ffc10 tx=0x7f77d800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:28.512 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.506+0000 7f77d67fc700 1 -- 172.21.15.102:0/3790399890 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f77dc01d690 con 0x7f77e4100740 2023-07-14T18:57:28.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:28.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:28.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:28.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:28.704 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.699+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi088=y;smithi102=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f77e4060a80 con 0x7f77d00389e0 2023-07-14T18:57:28.711 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.705+0000 7f77d67fc700 1 -- 172.21.15.102:0/3790399890 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f77e4060a80 con 0x7f77d00389e0 2023-07-14T18:57:28.711 INFO:teuthology.orchestra.run.smithi102.stdout:Scheduled mgr update... 2023-07-14T18:57:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77d00389e0 msgr2=0x7f77d003ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77d00389e0 0x7f77d003ae90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f77e40ffc10 tx=0x7f77d800a380 comp rx=0 tx=0).stop 2023-07-14T18:57:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 msgr2=0x7f77e419cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 0x7f77e419cd90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f77dc0062a0 tx=0x7f77dc004d00 comp rx=0 tx=0).stop 2023-07-14T18:57:28.713 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 shutdown_connections 2023-07-14T18:57:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77d00389e0 0x7f77d003ae90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77e4100740 0x7f77e419cd90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77e40febb0 0x7f77e4069550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.707+0000 7f77eb6d7700 1 --2- 172.21.15.102:0/3790399890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77e40ffdb0 0x7f77e419a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.708+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 >> 172.21.15.102:0/3790399890 conn(0x7f77e40fa1e0 msgr2=0x7f77e40fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.708+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 shutdown_connections 2023-07-14T18:57:28.714 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:57:28.708+0000 7f77eb6d7700 1 -- 172.21.15.102:0/3790399890 wait complete. 2023-07-14T18:57:29.063 DEBUG:teuthology.orchestra.run.smithi088:mgr.y> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.y.service 2023-07-14T18:57:29.066 DEBUG:teuthology.orchestra.run.smithi102:mgr.z> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.z.service 2023-07-14T18:57:29.068 INFO:tasks.cephadm:Deploying OSDs... 2023-07-14T18:57:29.069 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T18:57:29.069 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:57:29.089 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:57:29.090 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2023-07-14T18:57:29.096 INFO:journalctl@ceph.mgr.z.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T18:57:29.097 INFO:journalctl@ceph.mgr.y.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T18:57:29.150 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:57:29.151 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:29.151 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 380587 Links: 1 2023-07-14T18:57:29.151 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:29.151 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:29.151 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-07-14 18:56:40.685396760 +0000 2023-07-14T18:57:29.152 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-07-14 18:50:10.289206860 +0000 2023-07-14T18:57:29.152 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-07-14 18:50:10.289206860 +0000 2023-07-14T18:57:29.152 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-07-14T18:57:29.152 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:57:29.223 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-07-14T18:57:29.223 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-07-14T18:57:29.223 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000262939 s, 1.9 MB/s 2023-07-14T18:57:29.224 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:57:29.286 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2023-07-14T18:57:29.347 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:57:29.347 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:29.347 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 379166 Links: 1 2023-07-14T18:57:29.347 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:29.347 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:29.348 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-07-14 18:56:40.685396760 +0000 2023-07-14T18:57:29.348 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-07-14 18:50:10.781198877 +0000 2023-07-14T18:57:29.348 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-07-14 18:50:10.781198877 +0000 2023-07-14T18:57:29.348 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-07-14T18:57:29.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:57:29.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:29.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:29.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:29.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:29.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:29.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:29.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:29.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:57:29.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:29.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:29.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:29.418 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:29.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:29.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:29.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:29.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:29.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:29.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:57:29.419 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:29.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:29.421 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-07-14T18:57:29.421 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-07-14T18:57:29.421 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000247278 s, 2.1 MB/s 2023-07-14T18:57:29.422 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:57:29.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:29.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:29.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:29.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:29.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:29.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:29.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:29.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:57:29.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:29.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:29.484 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2023-07-14T18:57:29.545 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:57:29.545 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:29.545 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 377842 Links: 1 2023-07-14T18:57:29.546 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:29.546 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:29.546 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-07-14 18:56:40.685396760 +0000 2023-07-14T18:57:29.546 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-07-14 18:50:11.295190537 +0000 2023-07-14T18:57:29.546 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-07-14 18:50:11.295190537 +0000 2023-07-14T18:57:29.546 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-07-14T18:57:29.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:57:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-07-14T18:57:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-07-14T18:57:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.00025143 s, 2.0 MB/s 2023-07-14T18:57:29.617 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:57:29.679 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2023-07-14T18:57:29.740 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:57:29.740 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:29.741 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 6h/6d Inode: 381261 Links: 1 2023-07-14T18:57:29.741 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:29.741 INFO:teuthology.orchestra.run.smithi078.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:29.741 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2023-07-14 18:56:40.685396760 +0000 2023-07-14T18:57:29.741 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2023-07-14 18:50:11.802182311 +0000 2023-07-14T18:57:29.741 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2023-07-14 18:50:11.802182311 +0000 2023-07-14T18:57:29.741 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2023-07-14T18:57:29.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:57:29.810 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2023-07-14T18:57:29.811 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2023-07-14T18:57:29.811 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000241525 s, 2.1 MB/s 2023-07-14T18:57:29.812 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:57:29.874 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2023-07-14T18:57:29.874 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:57:29.894 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:57:29.895 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2023-07-14T18:57:29.954 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:57:29.955 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:29.955 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 375173 Links: 1 2023-07-14T18:57:29.955 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:29.955 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:29.956 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-14 18:57:15.625809456 +0000 2023-07-14T18:57:29.956 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-14 18:45:56.324693245 +0000 2023-07-14T18:57:29.956 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-14 18:45:56.324693245 +0000 2023-07-14T18:57:29.956 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-14T18:57:29.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:57:30.025 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-14T18:57:30.026 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-14T18:57:30.026 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000296475 s, 1.7 MB/s 2023-07-14T18:57:30.027 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:57:30.088 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2023-07-14T18:57:30.148 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:57:30.148 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:30.148 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 382198 Links: 1 2023-07-14T18:57:30.148 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:30.149 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:30.149 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-14 18:57:15.625809456 +0000 2023-07-14T18:57:30.149 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-14 18:45:56.854683048 +0000 2023-07-14T18:57:30.149 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-14 18:45:56.854683048 +0000 2023-07-14T18:57:30.149 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-14T18:57:30.150 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:57:30.216 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-14T18:57:30.217 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-14T18:57:30.217 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000234503 s, 2.2 MB/s 2023-07-14T18:57:30.218 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:57:30.278 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2023-07-14T18:57:30.337 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:57:30.337 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:30.337 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 375394 Links: 1 2023-07-14T18:57:30.337 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:30.338 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:30.338 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-14 18:57:15.625809456 +0000 2023-07-14T18:57:30.338 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-14 18:45:57.377672986 +0000 2023-07-14T18:57:30.338 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-14 18:45:57.377672986 +0000 2023-07-14T18:57:30.338 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-14T18:57:30.338 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:57:30.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:30 smithi102 ceph-mon[129280]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi088=y;smithi102=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:30.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:30 smithi102 ceph-mon[129280]: Saving service mgr spec with placement smithi078=x;smithi088=y;smithi102=z;count:3 2023-07-14T18:57:30.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:30 smithi102 ceph-mon[129280]: Deploying daemon mgr.z on smithi102 2023-07-14T18:57:30.383 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:30 smithi102 systemd[1]: Starting Ceph mgr.z for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T18:57:30.403 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:30 smithi088 ceph-mon[128973]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi088=y;smithi102=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:30.403 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:30 smithi088 ceph-mon[128973]: Saving service mgr spec with placement smithi078=x;smithi088=y;smithi102=z;count:3 2023-07-14T18:57:30.403 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:30 smithi088 ceph-mon[128973]: Deploying daemon mgr.z on smithi102 2023-07-14T18:57:30.405 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-14T18:57:30.405 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-14T18:57:30.405 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000244491 s, 2.1 MB/s 2023-07-14T18:57:30.406 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:57:30.465 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2023-07-14T18:57:30.524 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:57:30.524 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:30.525 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 6h/6d Inode: 383168 Links: 1 2023-07-14T18:57:30.525 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:30.525 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:30.525 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2023-07-14 18:57:15.625809456 +0000 2023-07-14T18:57:30.525 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2023-07-14 18:45:57.873663443 +0000 2023-07-14T18:57:30.525 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2023-07-14 18:45:57.873663443 +0000 2023-07-14T18:57:30.525 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: - 2023-07-14T18:57:30.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:57:30.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:30 smithi078 ceph-mon[127425]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi078=x;smithi088=y;smithi102=z", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:30.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:30 smithi078 ceph-mon[127425]: Saving service mgr spec with placement smithi078=x;smithi088=y;smithi102=z;count:3 2023-07-14T18:57:30.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:30 smithi078 ceph-mon[127425]: Deploying daemon mgr.z on smithi102 2023-07-14T18:57:30.592 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2023-07-14T18:57:30.592 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2023-07-14T18:57:30.592 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000244487 s, 2.1 MB/s 2023-07-14T18:57:30.593 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:57:30.652 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2023-07-14T18:57:30.653 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2023-07-14T18:57:30.673 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-07-14T18:57:30.673 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2023-07-14T18:57:30.723 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:30 smithi102 podman[131150]: 2023-07-14T18:57:30.723 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:30 smithi102 bash[131150]: 639857f581588c66dbffd9e644b646aa8e3afeba06c644e002a808efcb0dbe3e 2023-07-14T18:57:30.723 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:30 smithi102 ceph-mgr[131190]: -- 172.21.15.102:0/3334730880 <== mon.1 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e0182b1e0 con 0x564e01800800 2023-07-14T18:57:30.731 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-07-14T18:57:30.732 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:30.732 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 376350 Links: 1 2023-07-14T18:57:30.732 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:30.732 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:30.732 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2023-07-14 18:57:30.086804727 +0000 2023-07-14T18:57:30.732 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-07-14 18:45:46.650791675 +0000 2023-07-14T18:57:30.732 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-07-14 18:45:46.650791675 +0000 2023-07-14T18:57:30.733 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2023-07-14T18:57:30.733 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-07-14T18:57:30.802 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2023-07-14T18:57:30.802 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2023-07-14T18:57:30.802 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000249133 s, 2.1 MB/s 2023-07-14T18:57:30.803 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-07-14T18:57:30.860 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2023-07-14T18:57:30.917 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-07-14T18:57:30.918 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:30.918 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 382020 Links: 1 2023-07-14T18:57:30.918 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:30.918 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:30.919 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2023-07-14 18:57:30.087804725 +0000 2023-07-14T18:57:30.919 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-07-14 18:45:47.173791459 +0000 2023-07-14T18:57:30.919 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-07-14 18:45:47.173791459 +0000 2023-07-14T18:57:30.919 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2023-07-14T18:57:30.920 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-07-14T18:57:30.983 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:30 smithi102 systemd[1]: Started Ceph mgr.z for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T18:57:30.984 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:30.762+0000 7f47b9d45200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:57:30.984 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:30.848+0000 7f47b9d45200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:57:30.984 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:30.940+0000 7f47b9d45200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:57:30.985 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2023-07-14T18:57:30.986 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2023-07-14T18:57:30.986 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000202399 s, 2.5 MB/s 2023-07-14T18:57:30.987 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-07-14T18:57:31.045 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2023-07-14T18:57:31.103 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-07-14T18:57:31.103 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:31.103 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 384037 Links: 1 2023-07-14T18:57:31.104 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:31.104 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:31.104 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2023-07-14 18:57:30.087804725 +0000 2023-07-14T18:57:31.105 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-07-14 18:45:47.699791242 +0000 2023-07-14T18:57:31.105 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-07-14 18:45:47.699791242 +0000 2023-07-14T18:57:31.105 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2023-07-14T18:57:31.106 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-07-14T18:57:31.169 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2023-07-14T18:57:31.169 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2023-07-14T18:57:31.169 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000226954 s, 2.3 MB/s 2023-07-14T18:57:31.170 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-07-14T18:57:31.227 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2023-07-14T18:57:31.288 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-07-14T18:57:31.289 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-07-14T18:57:31.289 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 6h/6d Inode: 378639 Links: 1 2023-07-14T18:57:31.289 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-07-14T18:57:31.290 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2023-07-14T18:57:31.290 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2023-07-14 18:57:30.087804725 +0000 2023-07-14T18:57:31.290 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2023-07-14 18:45:48.221791026 +0000 2023-07-14T18:57:31.290 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2023-07-14 18:45:48.221791026 +0000 2023-07-14T18:57:31.291 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: - 2023-07-14T18:57:31.291 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-07-14T18:57:31.353 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:31.040+0000 7f47b9d45200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:57:31.353 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:31.223+0000 7f47b9d45200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:57:31.354 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-mon[129280]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:31.354 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.354 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.355 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.355 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:31.355 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:57:31.355 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:31.356 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:31.357 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2023-07-14T18:57:31.357 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2023-07-14T18:57:31.357 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000172125 s, 3.0 MB/s 2023-07-14T18:57:31.358 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-07-14T18:57:31.415 INFO:tasks.cephadm:Deploying osd.0 on smithi078 with /dev/vg_nvme/lv_4... 2023-07-14T18:57:31.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:57:31.420 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:31 smithi088 ceph-mon[128973]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:31.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:31.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:57:31.421 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:31.422 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:31.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:31 smithi078 ceph-mon[127425]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:31.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:31.446 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:31.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2023-07-14T18:57:31.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:31.447 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:31.632 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:31.545+0000 7f47b9d45200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:57:31.982 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:57:32.124 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:31.632+0000 7f47b9d45200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:57:32.125 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:31.759+0000 7f47b9d45200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:57:32.125 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:31.852+0000 7f47b9d45200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:57:32.272 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:32 smithi088 ceph-mon[128973]: Deploying daemon mgr.y on smithi088 2023-07-14T18:57:32.377 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:32 smithi078 ceph-mon[127425]: Deploying daemon mgr.y on smithi088 2023-07-14T18:57:32.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:32 smithi102 ceph-mon[129280]: Deploying daemon mgr.y on smithi088 2023-07-14T18:57:32.382 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:32 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:32.268+0000 7f47b9d45200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:57:32.547 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:32 smithi088 systemd[1]: Starting Ceph mgr.y for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T18:57:32.813 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:32 smithi088 podman[130438]: 2023-07-14T18:57:33.110 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:32 smithi088 bash[130438]: 0363a37ae0f0f2f9a6977ca8d5f025499e3fef0fa87b1055fccc3ea505ab987a 2023-07-14T18:57:33.111 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:32 smithi088 ceph-mgr[130478]: -- 172.21.15.88:0/293678519 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d8c28251e0 con 0x55d8c27fa800 2023-07-14T18:57:33.111 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:32 smithi088 systemd[1]: Started Ceph mgr.y for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T18:57:33.111 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:32 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:32.932+0000 7fe9d2342200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-07-14T18:57:33.111 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:33.013+0000 7fe9d2342200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-07-14T18:57:33.111 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:33.106+0000 7fe9d2342200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-07-14T18:57:33.125 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:32 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:32.837+0000 7f47b9d45200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:57:33.126 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:33.006+0000 7f47b9d45200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:57:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:33 smithi078 ceph-mon[127425]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:33.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-mon[129280]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:33.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:33.389 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:33.208+0000 7fe9d2342200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-07-14T18:57:33.390 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-mon[128973]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:33.390 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.390 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.390 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.390 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:33.390 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:33.597 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:57:33.597 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:57:33.597 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-07-14T18:57:33.597 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-07-14T18:57:33.597 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112675 s, 931 MB/s 2023-07-14T18:57:33.598 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-07-14T18:57:33.713 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:33.384+0000 7fe9d2342200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-07-14T18:57:34.038 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi078:vg_nvme/lv_4 2023-07-14T18:57:34.053 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:33.714+0000 7fe9d2342200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-07-14T18:57:34.053 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:33.814+0000 7fe9d2342200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-07-14T18:57:34.053 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:33.949+0000 7fe9d2342200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-07-14T18:57:34.054 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:34 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:34.048+0000 7fe9d2342200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-07-14T18:57:34.110 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:33.822+0000 7f47b9d45200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:57:34.110 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:33 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:33.926+0000 7f47b9d45200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:57:34.382 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:34.104+0000 7f47b9d45200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:57:34.630 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:57:34.779 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:34.398+0000 7f47b9d45200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:57:34.779 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:34.626+0000 7f47b9d45200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:57:34.963 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:34 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:34.492+0000 7fe9d2342200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-07-14T18:57:35.041 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:34.774+0000 7f47b9d45200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:57:35.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:35.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.191 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:35.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:35.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:35.192 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:35.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:35.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.382 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:35.035+0000 7f47b9d45200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:57:35.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:35.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:35.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:35.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:35.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:35.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:35.385 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.463 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:35.074+0000 7fe9d2342200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-07-14T18:57:35.463 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:35.244+0000 7fe9d2342200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-07-14T18:57:35.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:35.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:35.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:35.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-07-14T18:57:35.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-07-14T18:57:35.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:35.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:35.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 -- 172.21.15.78:0/382575610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900100810 msgr2=0x7f8900102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:35.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 --2- 172.21.15.78:0/382575610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900100810 0x7f8900102bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f88f0009990 tx=0x7f88f0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:57:35.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 -- 172.21.15.78:0/382575610 shutdown_connections 2023-07-14T18:57:35.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 --2- 172.21.15.78:0/382575610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900100810 0x7f8900102bf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:35.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 --2- 172.21.15.78:0/382575610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8900105a50 0x7f8900107e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:35.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 --2- 172.21.15.78:0/382575610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8900103130 0x7f8900105510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:35.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 -- 172.21.15.78:0/382575610 >> 172.21.15.78:0/382575610 conn(0x7f89000fa290 msgr2=0x7f89000fc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:35.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 -- 172.21.15.78:0/382575610 shutdown_connections 2023-07-14T18:57:35.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.988+0000 7f890779f700 1 -- 172.21.15.78:0/382575610 wait complete. 2023-07-14T18:57:35.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890779f700 1 Processor -- start 2023-07-14T18:57:35.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890779f700 1 -- start start 2023-07-14T18:57:35.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890779f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8900100810 0x7f8900100170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:35.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890779f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900103130 0x7f89000fe7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:35.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890779f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8900105a50 0x7f89000fed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:35.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890779f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000ff480 con 0x7f8900105a50 2023-07-14T18:57:35.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890779f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89000ff5f0 con 0x7f8900103130 2023-07-14T18:57:35.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890779f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f890019f990 con 0x7f8900100810 2023-07-14T18:57:35.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8900105a50 0x7f89000fed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:35.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8904d3a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900103130 0x7f89000fe7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:35.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8900105a50 0x7f89000fed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33574/0 (socket says 172.21.15.78:33574) 2023-07-14T18:57:35.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8904d3a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900103130 0x7f89000fe7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.78:56994/0 (socket says 172.21.15.78:56994) 2023-07-14T18:57:35.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 -- 172.21.15.78:0/1126096283 learned_addr learned my addr 172.21.15.78:0/1126096283 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:57:35.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f890553b700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8900100810 0x7f8900100170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:36.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 -- 172.21.15.78:0/1126096283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8900100810 msgr2=0x7f8900100170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:36.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8900100810 0x7f8900100170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:36.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 -- 172.21.15.78:0/1126096283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900103130 msgr2=0x7f89000fe7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:36.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900103130 0x7f89000fe7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:36.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 -- 172.21.15.78:0/1126096283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88f8009c40 con 0x7f8900105a50 2023-07-14T18:57:36.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8904d3a700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900103130 0x7f89000fe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:57:36.000 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f8905d3c700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8900105a50 0x7f89000fed00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f88f800c110 tx=0x7f88f800c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:36.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.989+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f8017070 con 0x7f8900105a50 2023-07-14T18:57:36.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.990+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88f00095f0 con 0x7f8900105a50 2023-07-14T18:57:36.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.990+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f88f800f040 con 0x7f8900105a50 2023-07-14T18:57:36.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.990+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88f800e050 con 0x7f8900105a50 2023-07-14T18:57:36.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.990+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f890019ff80 con 0x7f8900105a50 2023-07-14T18:57:36.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.990+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89000fbe00 con 0x7f8900105a50 2023-07-14T18:57:36.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.991+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 13) v1 ==== 45539+0+0 (secure 0 0 0) 0x7f88f8004ad0 con 0x7f8900105a50 2023-07-14T18:57:36.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.991+0000 7f88f67fc700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f88ec0389e0 0x7f88ec03ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:36.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.991+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f88f804b230 con 0x7f8900105a50 2023-07-14T18:57:36.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.991+0000 7f890553b700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f88ec0389e0 0x7f88ec03ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:36.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.992+0000 7f890553b700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f88ec0389e0 0x7f88ec03ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88f0006010 tx=0x7f88f0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:36.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:35.995+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f88f801b740 con 0x7f8900105a50 2023-07-14T18:57:36.061 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:35.735+0000 7f47b9d45200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:57:36.061 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:35 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:35.864+0000 7f47b9d45200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:57:36.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:36.195+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8900060a80 con 0x7f88ec0389e0 2023-07-14T18:57:36.264 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:36 smithi088 ceph-mon[128973]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:57:36.265 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:36 smithi088 ceph-mon[128973]: Reconfiguring daemon mgr.x on smithi078 2023-07-14T18:57:36.265 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:36.000+0000 7fe9d2342200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:57:36.265 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:36.090+0000 7fe9d2342200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-07-14T18:57:36.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:36 smithi102 ceph-mon[129280]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:57:36.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:36 smithi102 ceph-mon[129280]: Reconfiguring daemon mgr.x on smithi078 2023-07-14T18:57:36.382 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:36.055+0000 7f47b9d45200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:57:36.383 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:36.169+0000 7f47b9d45200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:57:36.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:36 smithi078 ceph-mon[127425]: Reconfiguring mgr.x (unknown last config time)... 2023-07-14T18:57:36.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:36 smithi078 ceph-mon[127425]: Reconfiguring daemon mgr.x on smithi078 2023-07-14T18:57:36.882 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:36.384+0000 7f47b9d45200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:57:36.915 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:36.260+0000 7fe9d2342200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-07-14T18:57:36.916 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:36.540+0000 7fe9d2342200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-07-14T18:57:36.916 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:36.766+0000 7fe9d2342200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-07-14T18:57:37.174 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:36.911+0000 7fe9d2342200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-07-14T18:57:37.175 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:37.175 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:37.175 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:57:37.175 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:57:37.175 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:37.175 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.176 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.176 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:37.176 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:37.176 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:37.176 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.295 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:37.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:37.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:57:37.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:57:37.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:37.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:37.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:37.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:37.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.382 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z[131166]: 2023-07-14T18:57:37.235+0000 7f47b9d45200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:57:37.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:37.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:37.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:57:37.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:57:37.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:37.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:37.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:37.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:37.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:37.463 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:37.170+0000 7fe9d2342200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-07-14T18:57:38.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:38.134+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 14) v1 ==== 90599+0+0 (secure 0 0 0) 0x7f88f8049fc0 con 0x7f8900105a50 2023-07-14T18:57:38.213 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:37 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:37.892+0000 7fe9d2342200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-07-14T18:57:38.213 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:38.023+0000 7fe9d2342200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-07-14T18:57:38.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-mon[128973]: Standby manager daemon z started 2023-07-14T18:57:38.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-mon[128973]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:57:38.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-mon[128973]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:57:38.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-mon[128973]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:57:38.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-mon[128973]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:57:38.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:38 smithi102 ceph-mon[129280]: Standby manager daemon z started 2023-07-14T18:57:38.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:38 smithi102 ceph-mon[129280]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:57:38.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:38 smithi102 ceph-mon[129280]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:57:38.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:38 smithi102 ceph-mon[129280]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:57:38.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:38 smithi102 ceph-mon[129280]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:57:38.549 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:38.215+0000 7fe9d2342200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-07-14T18:57:38.549 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:38.328+0000 7fe9d2342200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-07-14T18:57:38.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:38 smithi078 ceph-mon[127425]: Standby manager daemon z started 2023-07-14T18:57:38.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:38 smithi078 ceph-mon[127425]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2023-07-14T18:57:38.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:38 smithi078 ceph-mon[127425]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:57:38.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:38 smithi078 ceph-mon[127425]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2023-07-14T18:57:38.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:38 smithi078 ceph-mon[127425]: from='mgr.? 172.21.15.102:0/2085816748' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:57:38.963 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:38.544+0000 7fe9d2342200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-07-14T18:57:39.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:39 smithi088 ceph-mon[128973]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:39.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:39 smithi088 ceph-mon[128973]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-14T18:57:39.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:57:39.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:39 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1459580008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94b51247-9012-4bba-941b-2a524896715f"}]: dispatch 2023-07-14T18:57:39.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:39 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1459580008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94b51247-9012-4bba-941b-2a524896715f"}]': finished 2023-07-14T18:57:39.399 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:39 smithi088 ceph-mon[128973]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:57:39.400 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:39.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:39 smithi078 ceph-mon[127425]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:39.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:39 smithi078 ceph-mon[127425]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-14T18:57:39.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:57:39.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:39 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1459580008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94b51247-9012-4bba-941b-2a524896715f"}]: dispatch 2023-07-14T18:57:39.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:39 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1459580008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94b51247-9012-4bba-941b-2a524896715f"}]': finished 2023-07-14T18:57:39.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:39 smithi078 ceph-mon[127425]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:57:39.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:39.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:39 smithi102 ceph-mon[129280]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:39.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:39 smithi102 ceph-mon[129280]: mgrmap e14: x(active, since 68s), standbys: z 2023-07-14T18:57:39.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2023-07-14T18:57:39.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:39 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1459580008' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "94b51247-9012-4bba-941b-2a524896715f"}]: dispatch 2023-07-14T18:57:39.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:39 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1459580008' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "94b51247-9012-4bba-941b-2a524896715f"}]': finished 2023-07-14T18:57:39.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:39 smithi102 ceph-mon[129280]: osdmap e5: 1 total, 0 up, 1 in 2023-07-14T18:57:39.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:39.714 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 18:57:39 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y[130454]: 2023-07-14T18:57:39.394+0000 7fe9d2342200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-07-14T18:57:40.148 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:40.143+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mon.0 v2:172.21.15.78:3300/0 8 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f88f801b370 con 0x7f8900105a50 2023-07-14T18:57:40.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:40 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/292928802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:57:40.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:40 smithi088 ceph-mon[128973]: Standby manager daemon y started 2023-07-14T18:57:40.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:40 smithi088 ceph-mon[128973]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:57:40.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:40 smithi088 ceph-mon[128973]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:57:40.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:40 smithi088 ceph-mon[128973]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:57:40.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:40 smithi088 ceph-mon[128973]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:57:40.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:40 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/292928802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:57:40.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:40 smithi078 ceph-mon[127425]: Standby manager daemon y started 2023-07-14T18:57:40.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:40 smithi078 ceph-mon[127425]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:57:40.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:40 smithi078 ceph-mon[127425]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:57:40.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:40 smithi078 ceph-mon[127425]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:57:40.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:40 smithi078 ceph-mon[127425]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:57:40.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:40 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/292928802' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:57:40.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:40 smithi102 ceph-mon[129280]: Standby manager daemon y started 2023-07-14T18:57:40.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:40 smithi102 ceph-mon[129280]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2023-07-14T18:57:40.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:40 smithi102 ceph-mon[129280]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2023-07-14T18:57:40.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:40 smithi102 ceph-mon[129280]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2023-07-14T18:57:40.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:40 smithi102 ceph-mon[129280]: from='mgr.? 172.21.15.88:0/1991264695' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2023-07-14T18:57:41.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:41 smithi088 ceph-mon[128973]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:41.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:41 smithi088 ceph-mon[128973]: mgrmap e15: x(active, since 70s), standbys: z, y 2023-07-14T18:57:41.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:57:41.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:41 smithi078 ceph-mon[127425]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:41.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:41 smithi078 ceph-mon[127425]: mgrmap e15: x(active, since 70s), standbys: z, y 2023-07-14T18:57:41.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:57:41.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:41 smithi102 ceph-mon[129280]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:41.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:41 smithi102 ceph-mon[129280]: mgrmap e15: x(active, since 70s), standbys: z, y 2023-07-14T18:57:41.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2023-07-14T18:57:43.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:43 smithi088 ceph-mon[128973]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:43.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:43 smithi078 ceph-mon[127425]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:43.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:43 smithi102 ceph-mon[129280]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:45.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:45 smithi088 ceph-mon[128973]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:45.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:45 smithi078 ceph-mon[127425]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:45.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:45 smithi102 ceph-mon[129280]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:46.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:57:46.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:46.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:46 smithi088 ceph-mon[128973]: Deploying daemon osd.0 on smithi078 2023-07-14T18:57:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:57:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:46 smithi078 ceph-mon[127425]: Deploying daemon osd.0 on smithi078 2023-07-14T18:57:46.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-07-14T18:57:46.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:46.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:46 smithi102 ceph-mon[129280]: Deploying daemon osd.0 on smithi078 2023-07-14T18:57:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:47 smithi078 ceph-mon[127425]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:47.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:47 smithi088 ceph-mon[128973]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:47.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:47 smithi102 ceph-mon[129280]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:49.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:49 smithi088 ceph-mon[128973]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:49.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:49 smithi078 ceph-mon[127425]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:49.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:49 smithi102 ceph-mon[129280]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:51.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:50 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:51.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:50 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:51.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:50 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:51.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:50 smithi078 ceph-mon[127425]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:51.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:50 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:51.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:50 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:51.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:50 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:51.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:50 smithi102 ceph-mon[129280]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:51.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:50 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:51.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:50 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:51.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:50 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:51.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:50 smithi088 ceph-mon[128973]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:52.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.747+0000 7f88f67fc700 1 -- 172.21.15.78:0/1126096283 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8900060a80 con 0x7f88ec0389e0 2023-07-14T18:57:52.753 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 0 on host 'smithi078' 2023-07-14T18:57:52.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f88ec0389e0 msgr2=0x7f88ec03ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:52.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f88ec0389e0 0x7f88ec03ae90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f88f0006010 tx=0x7f88f0005960 comp rx=0 tx=0).stop 2023-07-14T18:57:52.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8900105a50 msgr2=0x7f89000fed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:52.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8900105a50 0x7f89000fed00 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f88f800c110 tx=0x7f88f800c420 comp rx=0 tx=0).stop 2023-07-14T18:57:52.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 shutdown_connections 2023-07-14T18:57:52.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f88ec0389e0 0x7f88ec03ae90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:52.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8900105a50 0x7f89000fed00 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:52.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8900100810 0x7f8900100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 --2- 172.21.15.78:0/1126096283 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8900103130 0x7f89000fe7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 >> 172.21.15.78:0/1126096283 conn(0x7f89000fa290 msgr2=0x7f8900101470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 shutdown_connections 2023-07-14T18:57:52.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:57:52.749+0000 7f890779f700 1 -- 172.21.15.78:0/1126096283 wait complete. 2023-07-14T18:57:53.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:53.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:53.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:53.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:52 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:53.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:53.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:53.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:53.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:52 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:53.150 DEBUG:teuthology.orchestra.run.smithi078:osd.0> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.0.service 2023-07-14T18:57:53.154 INFO:tasks.cephadm:Deploying osd.1 on smithi088 with /dev/vg_nvme/lv_4... 2023-07-14T18:57:53.154 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:57:53.181 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.181 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.181 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:53.181 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:53.181 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.182 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:53.182 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.182 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:53.182 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:52 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:57:53.183 INFO:journalctl@ceph.osd.0.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T18:57:53.698 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:57:54.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:53 smithi078 ceph-mon[127425]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:57:54.002 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:57:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-0[140506]: 2023-07-14T18:57:53.792+0000 7f1fe900f640 -1 osd.0 0 log_to_monitors true 2023-07-14T18:57:54.121 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:53 smithi088 ceph-mon[128973]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:57:54.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:53 smithi102 ceph-mon[129280]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-07-14T18:57:55.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:54 smithi102 ceph-mon[129280]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:57:55.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:54 smithi102 ceph-mon[129280]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:57:55.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:54 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:55.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:54 smithi102 ceph-mon[129280]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:57:55.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:54 smithi102 ceph-mon[129280]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:55.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:54 smithi102 ceph-mon[129280]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T18:57:55.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:54 smithi102 ceph-mon[129280]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:57:55.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:54 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:55.148 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-0[140506]: 2023-07-14T18:57:54.856+0000 7f1fde687700 -1 osd.0 0 waiting for initial osdmap 2023-07-14T18:57:55.148 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-0[140506]: 2023-07-14T18:57:54.864+0000 7f1fd7c78700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:57:55.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-mon[127425]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:57:55.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-mon[127425]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:57:55.149 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:55.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-mon[127425]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:57:55.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-mon[127425]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:55.150 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-mon[127425]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T18:57:55.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-mon[127425]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:57:55.151 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:55.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:54 smithi088 ceph-mon[128973]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-07-14T18:57:55.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:54 smithi088 ceph-mon[128973]: osdmap e6: 1 total, 0 up, 1 in 2023-07-14T18:57:55.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:54 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:55.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:54 smithi088 ceph-mon[128973]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:57:55.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:54 smithi088 ceph-mon[128973]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:55.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:54 smithi088 ceph-mon[128973]: from='osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T18:57:55.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:54 smithi088 ceph-mon[128973]: osdmap e7: 1 total, 0 up, 1 in 2023-07-14T18:57:55.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:54 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:55.339 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:57:55.339 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:57:55.339 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-14T18:57:55.340 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-14T18:57:55.340 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0108119 s, 970 MB/s 2023-07-14T18:57:55.340 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-14T18:57:55.735 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_4 2023-07-14T18:57:56.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:56.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:57:56.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:56.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:56.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:55 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:56.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:56.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:57:56.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:56.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:56.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:55 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:56.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:56.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:57:56.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:56.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:57:56.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:57:56.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:55 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:56.324 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:57:57.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:56 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T18:57:57.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:56 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T18:57:57.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:56 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi078 2023-07-14T18:57:57.112 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:56 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi078 to 17168M 2023-07-14T18:57:57.113 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:56 smithi088 ceph-mon[128973]: osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] boot 2023-07-14T18:57:57.113 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:56 smithi088 ceph-mon[128973]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:57:57.113 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:57.113 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:56 smithi088 ceph-mon[128973]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:56 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T18:57:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:56 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T18:57:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:56 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi078 2023-07-14T18:57:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:56 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi078 to 17168M 2023-07-14T18:57:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:56 smithi078 ceph-mon[127425]: osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] boot 2023-07-14T18:57:57.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:56 smithi078 ceph-mon[127425]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:57:57.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:57.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:56 smithi078 ceph-mon[127425]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:57.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:56 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T18:57:57.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:56 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T18:57:57.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:56 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi078 2023-07-14T18:57:57.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:56 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi078 to 17168M 2023-07-14T18:57:57.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:56 smithi102 ceph-mon[129280]: osd.0 [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] boot 2023-07-14T18:57:57.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:56 smithi102 ceph-mon[129280]: osdmap e8: 1 total, 1 up, 1 in 2023-07-14T18:57:57.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-07-14T18:57:57.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:56 smithi102 ceph-mon[129280]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-07-14T18:57:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 -- 172.21.15.88:0/1001948244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c100ec0 msgr2=0x7f0c9c101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 --2- 172.21.15.88:0/1001948244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c100ec0 0x7f0c9c101310 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0c90009990 tx=0x7f0c90009ca0 comp rx=0 tx=0).stop 2023-07-14T18:57:57.690 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 -- 172.21.15.88:0/1001948244 shutdown_connections 2023-07-14T18:57:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 --2- 172.21.15.88:0/1001948244 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c9c101850 0x7f0c9c108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 --2- 172.21.15.88:0/1001948244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9c0ffcc0 0x7f0c9c1000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 --2- 172.21.15.88:0/1001948244 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c100ec0 0x7f0c9c101310 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 -- 172.21.15.88:0/1001948244 >> 172.21.15.88:0/1001948244 conn(0x7f0c9c0fb2d0 msgr2=0x7f0c9c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:57:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 -- 172.21.15.88:0/1001948244 shutdown_connections 2023-07-14T18:57:57.691 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.684+0000 7f0ca2114700 1 -- 172.21.15.88:0/1001948244 wait complete. 2023-07-14T18:57:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.685+0000 7f0ca2114700 1 Processor -- start 2023-07-14T18:57:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.685+0000 7f0ca2114700 1 -- start start 2023-07-14T18:57:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.685+0000 7f0ca2114700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9c0ffcc0 0x7f0c9c195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.685+0000 7f0ca2114700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c9c100ec0 0x7f0c9c195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.685+0000 7f0ca2114700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c101850 0x7f0c9c19ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:57.693 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.685+0000 7f0ca2114700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c19d4a0 con 0x7f0c9c100ec0 2023-07-14T18:57:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.685+0000 7f0ca2114700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c19d5e0 con 0x7f0c9c101850 2023-07-14T18:57:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.685+0000 7f0ca2114700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c9c19d770 con 0x7f0c9c0ffcc0 2023-07-14T18:57:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c101850 0x7f0c9c19ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9b7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9c0ffcc0 0x7f0c9c195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c101850 0x7f0c9c19ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59828/0 (socket says 172.21.15.88:59828) 2023-07-14T18:57:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 -- 172.21.15.88:0/2618119991 learned_addr learned my addr 172.21.15.88:0/2618119991 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T18:57:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9affd700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c9c100ec0 0x7f0c9c195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:57.694 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 -- 172.21.15.88:0/2618119991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9c0ffcc0 msgr2=0x7f0c9c195910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9c0ffcc0 0x7f0c9c195910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 -- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c9c100ec0 msgr2=0x7f0c9c195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:57:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c9c100ec0 0x7f0c9c195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:57:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 -- 172.21.15.88:0/2618119991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c900095f0 con 0x7f0c9c101850 2023-07-14T18:57:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c9bfff700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c101850 0x7f0c9c19ced0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0c94007ee0 tx=0x7f0c9400d3b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c98ff9700 1 -- 172.21.15.88:0/2618119991 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94017070 con 0x7f0c9c101850 2023-07-14T18:57:57.695 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c98ff9700 1 -- 172.21.15.88:0/2618119991 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0c9400f040 con 0x7f0c9c101850 2023-07-14T18:57:57.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0c98ff9700 1 -- 172.21.15.88:0/2618119991 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c94005550 con 0x7f0c9c101850 2023-07-14T18:57:57.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c9c19da50 con 0x7f0c9c101850 2023-07-14T18:57:57.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.686+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c9c19df00 con 0x7f0c9c101850 2023-07-14T18:57:57.696 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.687+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9c106c50 con 0x7f0c9c101850 2023-07-14T18:57:57.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.688+0000 7f0c98ff9700 1 -- 172.21.15.88:0/2618119991 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f0c9400db20 con 0x7f0c9c101850 2023-07-14T18:57:57.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.689+0000 7f0c98ff9700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c840a1140 0x7f0c840a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:57:57.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.689+0000 7f0c98ff9700 1 -- 172.21.15.88:0/2618119991 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f0c940ca5b0 con 0x7f0c9c101850 2023-07-14T18:57:57.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.689+0000 7f0c9b7fe700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c840a1140 0x7f0c840a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:57:57.697 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.690+0000 7f0c9b7fe700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c840a1140 0x7f0c840a35f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c100d20 tx=0x7f0c8c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:57:57.699 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.694+0000 7f0c98ff9700 1 -- 172.21.15.88:0/2618119991 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c940956d0 con 0x7f0c9c101850 2023-07-14T18:57:57.893 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:57:57.888+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0c9c060a80 con 0x7f0c840a1140 2023-07-14T18:57:58.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:57 smithi088 ceph-mon[128973]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:57:58.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:57 smithi078 ceph-mon[127425]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:57:58.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:57 smithi102 ceph-mon[129280]: osdmap e9: 1 total, 1 up, 1 in 2023-07-14T18:57:59.075 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:58 smithi088 ceph-mon[128973]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:59.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:58 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:57:59.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:58 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:57:59.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:58 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:59.076 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:57:58 smithi088 ceph-mon[128973]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:57:59.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:58 smithi078 ceph-mon[127425]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:59.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:58 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:57:59.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:58 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:57:59.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:58 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:59.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:57:58 smithi078 ceph-mon[127425]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:57:59.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:58 smithi102 ceph-mon[129280]: from='client.24152 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:57:59.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:58 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:57:59.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:58 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:57:59.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:58 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:57:59.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:57:58 smithi102 ceph-mon[129280]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:01.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:01 smithi102 ceph-mon[129280]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:01.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:01 smithi102 ceph-mon[129280]: from='client.? 172.21.15.88:0/1992168663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]: dispatch 2023-07-14T18:58:01.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:01 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]: dispatch 2023-07-14T18:58:01.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:01 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]': finished 2023-07-14T18:58:01.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:01 smithi102 ceph-mon[129280]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:58:01.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:01.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:01 smithi102 ceph-mon[129280]: from='client.? 172.21.15.88:0/638391388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:01.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:01 smithi088 ceph-mon[128973]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:01.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:01 smithi088 ceph-mon[128973]: from='client.? 172.21.15.88:0/1992168663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]: dispatch 2023-07-14T18:58:01.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:01 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]: dispatch 2023-07-14T18:58:01.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:01 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]': finished 2023-07-14T18:58:01.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:01 smithi088 ceph-mon[128973]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:58:01.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:01.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:01 smithi088 ceph-mon[128973]: from='client.? 172.21.15.88:0/638391388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:01 smithi078 ceph-mon[127425]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:01 smithi078 ceph-mon[127425]: from='client.? 172.21.15.88:0/1992168663' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]: dispatch 2023-07-14T18:58:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:01 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]: dispatch 2023-07-14T18:58:01.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:01 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5f66a20-1101-4355-8ca4-9d3de118d5c8"}]': finished 2023-07-14T18:58:01.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:01 smithi078 ceph-mon[127425]: osdmap e10: 2 total, 1 up, 2 in 2023-07-14T18:58:01.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:01.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:01 smithi078 ceph-mon[127425]: from='client.? 172.21.15.88:0/638391388' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:03.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:03 smithi102 ceph-mon[129280]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:03.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:03 smithi088 ceph-mon[128973]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:03.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:03 smithi078 ceph-mon[127425]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:05.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:05 smithi102 ceph-mon[129280]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:05.439 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:05 smithi088 ceph-mon[128973]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:05.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:05 smithi078 ceph-mon[127425]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:06.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:06 smithi078 ceph-mon[127425]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:06.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:06 smithi102 ceph-mon[129280]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:06.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:06 smithi088 ceph-mon[128973]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:07.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:07 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:58:07.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:07 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:07.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:07 smithi078 ceph-mon[127425]: Deploying daemon osd.1 on smithi088 2023-07-14T18:58:07.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:58:07.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:07.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:07 smithi102 ceph-mon[129280]: Deploying daemon osd.1 on smithi088 2023-07-14T18:58:07.906 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:07 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-07-14T18:58:07.907 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:07 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:07.907 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:07 smithi088 ceph-mon[128973]: Deploying daemon osd.1 on smithi088 2023-07-14T18:58:08.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:08 smithi102 ceph-mon[129280]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:08.904 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:08 smithi088 ceph-mon[128973]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:09.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:08 smithi078 ceph-mon[127425]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:11.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:11 smithi102 ceph-mon[129280]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:11.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:11 smithi088 ceph-mon[128973]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:11 smithi078 ceph-mon[127425]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:12.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:12.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:12.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:12.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:12 smithi088 ceph-mon[128973]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:13.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:13.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:13.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:13.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:12 smithi078 ceph-mon[127425]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:13.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:13.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:13.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:13.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:12 smithi102 ceph-mon[129280]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:14.707 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.703+0000 7f0c98ff9700 1 -- 172.21.15.88:0/2618119991 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0c9c060a80 con 0x7f0c840a1140 2023-07-14T18:58:14.708 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 1 on host 'smithi088' 2023-07-14T18:58:14.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c840a1140 msgr2=0x7f0c840a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:14.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c840a1140 0x7f0c840a35f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0c9c100d20 tx=0x7f0c8c00b410 comp rx=0 tx=0).stop 2023-07-14T18:58:14.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c101850 msgr2=0x7f0c9c19ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:14.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c101850 0x7f0c9c19ced0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0c94007ee0 tx=0x7f0c9400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:58:14.710 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 shutdown_connections 2023-07-14T18:58:14.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c840a1140 0x7f0c840a35f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:14.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c9c101850 0x7f0c9c19ced0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:14.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c9c0ffcc0 0x7f0c9c195910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:14.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.705+0000 7f0ca2114700 1 --2- 172.21.15.88:0/2618119991 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c9c100ec0 0x7f0c9c195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:14.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.706+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 >> 172.21.15.88:0/2618119991 conn(0x7f0c9c0fb2d0 msgr2=0x7f0c9c0fd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:58:14.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.706+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 shutdown_connections 2023-07-14T18:58:14.711 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:58:14.706+0000 7f0ca2114700 1 -- 172.21.15.88:0/2618119991 wait complete. 2023-07-14T18:58:14.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:14.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:14.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:14.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:14.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:14.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:14 smithi088 ceph-mon[128973]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:15.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:15.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:15.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:15.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:15.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:15.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:14 smithi078 ceph-mon[127425]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:15.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:15.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:15.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:15.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:15.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:15.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:14 smithi102 ceph-mon[129280]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:15.314 DEBUG:teuthology.orchestra.run.smithi088:osd.1> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.1.service 2023-07-14T18:58:15.316 INFO:tasks.cephadm:Deploying osd.2 on smithi102 with /dev/vg_nvme/lv_4... 2023-07-14T18:58:15.317 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-07-14T18:58:15.346 INFO:journalctl@ceph.osd.1.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T18:58:15.670 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:15 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-1[134087]: 2023-07-14T18:58:15.592+0000 7fa9228b9640 -1 osd.1 0 log_to_monitors true 2023-07-14T18:58:15.885 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:58:15.928 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:15 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:15.928 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:15 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:15.928 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:15 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:15.929 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:15 smithi088 ceph-mon[128973]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:58:15.929 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:15 smithi088 ceph-mon[128973]: from='osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:58:16.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:15 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:16.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:15 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:16.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:15 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:16.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:15 smithi078 ceph-mon[127425]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:58:16.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:15 smithi078 ceph-mon[127425]: from='osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:58:16.074 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:15 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:16.074 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:15 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:16.074 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:15 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:16.074 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:15 smithi102 ceph-mon[129280]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:58:16.075 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:15 smithi102 ceph-mon[129280]: from='osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-07-14T18:58:16.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:16 smithi088 ceph-mon[128973]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:16.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:16 smithi088 ceph-mon[128973]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:58:16.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:16 smithi088 ceph-mon[128973]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:58:16.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:16 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:16.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:16 smithi088 ceph-mon[128973]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:58:16.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:16 smithi088 ceph-mon[128973]: from='osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:58:17.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:16 smithi078 ceph-mon[127425]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:17.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:16 smithi078 ceph-mon[127425]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:58:17.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:16 smithi078 ceph-mon[127425]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:58:17.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:16 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:17.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:16 smithi078 ceph-mon[127425]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:58:17.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:16 smithi078 ceph-mon[127425]: from='osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:58:17.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:16 smithi102 ceph-mon[129280]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:17.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:16 smithi102 ceph-mon[129280]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-07-14T18:58:17.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:16 smithi102 ceph-mon[129280]: osdmap e11: 2 total, 1 up, 2 in 2023-07-14T18:58:17.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:16 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:17.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:16 smithi102 ceph-mon[129280]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:58:17.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:16 smithi102 ceph-mon[129280]: from='osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:58:17.247 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-07-14T18:58:17.247 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-07-14T18:58:17.248 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2023-07-14T18:58:17.248 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2023-07-14T18:58:17.248 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0158907 s, 660 MB/s 2023-07-14T18:58:17.248 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 2023-07-14T18:58:17.248 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2023-07-14T18:58:17.734 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi102:vg_nvme/lv_4 2023-07-14T18:58:17.963 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:17 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-1[134087]: 2023-07-14T18:58:17.548+0000 7fa917f31700 -1 osd.1 0 waiting for initial osdmap 2023-07-14T18:58:17.964 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:17 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-1[134087]: 2023-07-14T18:58:17.551+0000 7fa910d21700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:58:18.287 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:58:18.750 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T18:58:18.751 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:58:18.751 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:18.752 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:18.752 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi088 2023-07-14T18:58:18.752 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:18.752 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:18.753 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:18.753 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-14T18:58:18.753 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:18.754 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:18.754 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:18.754 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:18.754 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:18 smithi102 ceph-mon[129280]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:18.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T18:58:18.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:58:18.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:18.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:18.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi088 2023-07-14T18:58:18.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:18.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:18.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:18.966 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-14T18:58:18.966 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:18.966 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:18.966 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:18.966 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:18.967 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:18 smithi088 ceph-mon[128973]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:19.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T18:58:19.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: osdmap e12: 2 total, 1 up, 2 in 2023-07-14T18:58:19.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:19.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:19.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi088 2023-07-14T18:58:19.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:19.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:19.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:19.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi088 to 17168M 2023-07-14T18:58:19.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:19.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:19.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:19.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:19.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:18 smithi078 ceph-mon[127425]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-07-14T18:58:19.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 -- 172.21.15.102:0/1768910482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fe380 msgr2=0x7f99b80fe790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:19.339 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/1768910482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fe380 0x7f99b80fe790 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f99ac009990 tx=0x7f99ac009ca0 comp rx=0 tx=0).stop 2023-07-14T18:58:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 -- 172.21.15.102:0/1768910482 shutdown_connections 2023-07-14T18:58:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/1768910482 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99b80fff10 0x7f99b81067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/1768910482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fe380 0x7f99b80fe790 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/1768910482 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b80ff580 0x7f99b80ff9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 -- 172.21.15.102:0/1768910482 >> 172.21.15.102:0/1768910482 conn(0x7f99b80f99b0 msgr2=0x7f99b80fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:58:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 -- 172.21.15.102:0/1768910482 shutdown_connections 2023-07-14T18:58:19.340 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.332+0000 7f99bd3e2700 1 -- 172.21.15.102:0/1768910482 wait complete. 2023-07-14T18:58:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99bd3e2700 1 Processor -- start 2023-07-14T18:58:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99bd3e2700 1 -- start start 2023-07-14T18:58:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99bd3e2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99b80fe380 0x7f99b81958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:58:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99bd3e2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b80ff580 0x7f99b8195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:58:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99bd3e2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fff10 0x7f99b819ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:58:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99bd3e2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b819d460 con 0x7f99b80fe380 2023-07-14T18:58:19.341 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99bd3e2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b819d5a0 con 0x7f99b80ff580 2023-07-14T18:58:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99bd3e2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b819d730 con 0x7f99b80fff10 2023-07-14T18:58:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99b77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fff10 0x7f99b819ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:58:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99b77fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fff10 0x7f99b819ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52060/0 (socket says 172.21.15.102:52060) 2023-07-14T18:58:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99b77fe700 1 -- 172.21.15.102:0/2474222415 learned_addr learned my addr 172.21.15.102:0/2474222415 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T18:58:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99b6ffd700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99b80fe380 0x7f99b81958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:58:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.333+0000 7f99b67fc700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b80ff580 0x7f99b8195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:58:19.342 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99b77fe700 1 -- 172.21.15.102:0/2474222415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b80ff580 msgr2=0x7f99b8195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99b77fe700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b80ff580 0x7f99b8195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99b77fe700 1 -- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99b80fe380 msgr2=0x7f99b81958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99b77fe700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99b80fe380 0x7f99b81958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99b77fe700 1 -- 172.21.15.102:0/2474222415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ac0095f0 con 0x7f99b80fff10 2023-07-14T18:58:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99b67fc700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b80ff580 0x7f99b8195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:58:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99b77fe700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fff10 0x7f99b819ce90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f99b0007d40 tx=0x7f99b000a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:58:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f999ffff700 1 -- 172.21.15.102:0/2474222415 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b000ad60 con 0x7f99b80fff10 2023-07-14T18:58:19.343 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f999ffff700 1 -- 172.21.15.102:0/2474222415 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f99b0007e90 con 0x7f99b80fff10 2023-07-14T18:58:19.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b819da10 con 0x7f99b80fff10 2023-07-14T18:58:19.344 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f999ffff700 1 -- 172.21.15.102:0/2474222415 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b00115e0 con 0x7f99b80fff10 2023-07-14T18:58:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.334+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b819dec0 con 0x7f99b80fff10 2023-07-14T18:58:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.335+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b8109a10 con 0x7f99b80fff10 2023-07-14T18:58:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.336+0000 7f999ffff700 1 -- 172.21.15.102:0/2474222415 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7f99b0012420 con 0x7f99b80fff10 2023-07-14T18:58:19.345 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.336+0000 7f999ffff700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f99a00a10c0 0x7f99a00a3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:58:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.337+0000 7f999ffff700 1 -- 172.21.15.102:0/2474222415 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f99b00cb6d0 con 0x7f99b80fff10 2023-07-14T18:58:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.337+0000 7f99b6ffd700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f99a00a10c0 0x7f99a00a3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:58:19.346 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.337+0000 7f99b6ffd700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f99a00a10c0 0x7f99a00a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b5c0 tx=0x7f99ac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:58:19.347 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.340+0000 7f999ffff700 1 -- 172.21.15.102:0/2474222415 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f99b0096670 con 0x7f99b80fff10 2023-07-14T18:58:19.539 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:19.533+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f99b81059d0 con 0x7f99a00a10c0 2023-07-14T18:58:19.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:19 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T18:58:19.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:19 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T18:58:19.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:19 smithi102 ceph-mon[129280]: osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] boot 2023-07-14T18:58:19.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:19 smithi102 ceph-mon[129280]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:58:19.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:19.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:58:19.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:58:19.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:19.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:19 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T18:58:19.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:19 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T18:58:19.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:19 smithi088 ceph-mon[128973]: osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] boot 2023-07-14T18:58:19.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:19 smithi088 ceph-mon[128973]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:58:19.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:19.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:58:19.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:58:19.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:20.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:19 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T18:58:20.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:19 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T18:58:20.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:19 smithi078 ceph-mon[127425]: osd.1 [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] boot 2023-07-14T18:58:20.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:19 smithi078 ceph-mon[127425]: osdmap e13: 2 total, 2 up, 2 in 2023-07-14T18:58:20.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-07-14T18:58:20.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:58:20.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:58:20.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:20.609 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:20 smithi102 ceph-mon[129280]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:58:20.609 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:20 smithi102 ceph-mon[129280]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:20.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:20 smithi088 ceph-mon[128973]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:58:20.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:20 smithi088 ceph-mon[128973]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:21.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:20 smithi078 ceph-mon[127425]: from='client.24160 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:58:21.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:20 smithi078 ceph-mon[127425]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:21.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:21 smithi102 ceph-mon[129280]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:58:21.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:21 smithi088 ceph-mon[128973]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:58:22.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:21 smithi078 ceph-mon[127425]: osdmap e14: 2 total, 2 up, 2 in 2023-07-14T18:58:22.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:22 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/980040832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]: dispatch 2023-07-14T18:58:22.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:22 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]: dispatch 2023-07-14T18:58:22.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:22 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]': finished 2023-07-14T18:58:22.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:22 smithi102 ceph-mon[129280]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:58:22.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:22 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:22.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:22 smithi102 ceph-mon[129280]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:22.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:22 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/1476700301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:22.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:22 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/980040832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]: dispatch 2023-07-14T18:58:22.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:22 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]: dispatch 2023-07-14T18:58:22.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:22 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]': finished 2023-07-14T18:58:22.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:22 smithi088 ceph-mon[128973]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:58:22.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:22 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:22.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:22 smithi088 ceph-mon[128973]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:22.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:22 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/1476700301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:23.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:22 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/980040832' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]: dispatch 2023-07-14T18:58:23.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:22 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]: dispatch 2023-07-14T18:58:23.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:22 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "180e6f44-2edd-427f-ac80-850d9e605f23"}]': finished 2023-07-14T18:58:23.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:22 smithi078 ceph-mon[127425]: osdmap e15: 3 total, 2 up, 3 in 2023-07-14T18:58:23.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:22 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:23.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:22 smithi078 ceph-mon[127425]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:23.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:22 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/1476700301' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:25.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:25 smithi102 ceph-mon[129280]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:25.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:25 smithi088 ceph-mon[128973]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:25.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:25 smithi078 ceph-mon[127425]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:26.840 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:26 smithi102 ceph-mon[129280]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:26.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:26 smithi088 ceph-mon[128973]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:27.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:26 smithi078 ceph-mon[127425]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:29.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:29 smithi102 ceph-mon[129280]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:29.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:58:29.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:29.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:29 smithi102 ceph-mon[129280]: Deploying daemon osd.2 on smithi102 2023-07-14T18:58:29.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:29 smithi088 ceph-mon[128973]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:29.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:58:29.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:29.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:29 smithi088 ceph-mon[128973]: Deploying daemon osd.2 on smithi102 2023-07-14T18:58:29.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:29 smithi078 ceph-mon[127425]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:29.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-07-14T18:58:29.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:29.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:29 smithi078 ceph-mon[127425]: Deploying daemon osd.2 on smithi102 2023-07-14T18:58:31.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:31 smithi102 ceph-mon[129280]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:31 smithi088 ceph-mon[128973]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:31.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:31 smithi078 ceph-mon[127425]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:33.491 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:33 smithi102 ceph-mon[129280]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:33.491 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:33 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:33.491 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:33 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:33.491 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:33 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:33.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:33 smithi088 ceph-mon[128973]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:33.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:33 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:33.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:33 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:33.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:33 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:33.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:33 smithi078 ceph-mon[127425]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:33.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:33.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:33.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:33 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:35.045 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.040+0000 7f999ffff700 1 -- 172.21.15.102:0/2474222415 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f99b81059d0 con 0x7f99a00a10c0 2023-07-14T18:58:35.046 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 2 on host 'smithi102' 2023-07-14T18:58:35.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f99a00a10c0 msgr2=0x7f99a00a3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:35.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f99a00a10c0 0x7f99a00a3570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f99ac00b5c0 tx=0x7f99ac005fb0 comp rx=0 tx=0).stop 2023-07-14T18:58:35.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fff10 msgr2=0x7f99b819ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:35.047 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fff10 0x7f99b819ce90 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f99b0007d40 tx=0x7f99b000a340 comp rx=0 tx=0).stop 2023-07-14T18:58:35.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 shutdown_connections 2023-07-14T18:58:35.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f99a00a10c0 0x7f99a00a3570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:35.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f99b80fff10 0x7f99b819ce90 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:35.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f99b80fe380 0x7f99b81958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:35.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 --2- 172.21.15.102:0/2474222415 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f99b80ff580 0x7f99b8195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:35.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 >> 172.21.15.102:0/2474222415 conn(0x7f99b80f99b0 msgr2=0x7f99b80fbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:58:35.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 shutdown_connections 2023-07-14T18:58:35.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:58:35.042+0000 7f99bd3e2700 1 -- 172.21.15.102:0/2474222415 wait complete. 2023-07-14T18:58:35.462 DEBUG:teuthology.orchestra.run.smithi102:osd.2> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.2.service 2023-07-14T18:58:35.464 INFO:tasks.cephadm:Deploying osd.3 on smithi078 with /dev/vg_nvme/lv_3... 2023-07-14T18:58:35.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:58:35.496 INFO:journalctl@ceph.osd.2.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T18:58:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:35.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:35.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:35.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:35.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:35.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:35.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:35.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:35.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:35.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:35.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:35.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:36.018 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:58:36.559 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-2[135106]: 2023-07-14T18:58:36.330+0000 7f0154501640 -1 osd.2 0 log_to_monitors true 2023-07-14T18:58:36.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:36 smithi078 ceph-mon[127425]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:36.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:36 smithi078 ceph-mon[127425]: from='osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:58:36.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:36 smithi078 ceph-mon[127425]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:58:36.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:36 smithi102 ceph-mon[129280]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:36.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:36 smithi102 ceph-mon[129280]: from='osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:58:36.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:36 smithi102 ceph-mon[129280]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:58:36.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:36 smithi088 ceph-mon[128973]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:36.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:36 smithi088 ceph-mon[128973]: from='osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:58:36.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:36 smithi088 ceph-mon[128973]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-07-14T18:58:37.491 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:58:37.491 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:58:37.491 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-07-14T18:58:37.491 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-07-14T18:58:37.492 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0129802 s, 808 MB/s 2023-07-14T18:58:37.492 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-07-14T18:58:37.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:37 smithi078 ceph-mon[127425]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:58:37.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:37 smithi078 ceph-mon[127425]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:58:37.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:37 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:37.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:37 smithi078 ceph-mon[127425]: from='osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:58:37.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:37 smithi078 ceph-mon[127425]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:58:37.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi078:vg_nvme/lv_3 2023-07-14T18:58:37.882 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:37 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-2[135106]: 2023-07-14T18:58:37.568+0000 7f0149b79700 -1 osd.2 0 waiting for initial osdmap 2023-07-14T18:58:37.882 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:37 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-2[135106]: 2023-07-14T18:58:37.571+0000 7f014396b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:58:37.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:37 smithi102 ceph-mon[129280]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:58:37.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:37 smithi102 ceph-mon[129280]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:58:37.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:37 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:37.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:37 smithi102 ceph-mon[129280]: from='osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:58:37.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:37 smithi102 ceph-mon[129280]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:58:37.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:37 smithi088 ceph-mon[128973]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-07-14T18:58:37.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:37 smithi088 ceph-mon[128973]: osdmap e16: 3 total, 2 up, 3 in 2023-07-14T18:58:37.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:37 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:37.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:37 smithi088 ceph-mon[128973]: from='osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:58:37.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:37 smithi088 ceph-mon[128973]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:58:38.420 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:58:38.836 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T18:58:38.836 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:58:38.836 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:38.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:38.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi102 2023-07-14T18:58:38.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:38.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi102 to 17168M 2023-07-14T18:58:38.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:38.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:38.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:38 smithi078 ceph-mon[127425]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:38.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T18:58:38.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:58:38.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:38.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:38.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi102 2023-07-14T18:58:38.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:38.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi102 to 17168M 2023-07-14T18:58:38.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:38.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:38.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:38 smithi102 ceph-mon[129280]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:38.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T18:58:38.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: osdmap e17: 3 total, 2 up, 3 in 2023-07-14T18:58:38.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:38.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:38.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi102 2023-07-14T18:58:38.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:38.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi102 to 17168M 2023-07-14T18:58:38.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:38.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:38.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:38.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:38 smithi088 ceph-mon[128973]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-07-14T18:58:39.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.763+0000 7fcf30a31700 1 -- 172.21.15.78:0/4045207786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1030c0 msgr2=0x7fcf2c1054a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.763+0000 7fcf30a31700 1 --2- 172.21.15.78:0/4045207786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1030c0 0x7fcf2c1054a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcf14009990 tx=0x7fcf14009ca0 comp rx=0 tx=0).stop 2023-07-14T18:58:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.763+0000 7fcf30a31700 1 -- 172.21.15.78:0/4045207786 shutdown_connections 2023-07-14T18:58:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.763+0000 7fcf30a31700 1 --2- 172.21.15.78:0/4045207786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf2c1007a0 0x7fcf2c102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.763+0000 7fcf30a31700 1 --2- 172.21.15.78:0/4045207786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2c1059e0 0x7fcf2c107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.763+0000 7fcf30a31700 1 --2- 172.21.15.78:0/4045207786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1030c0 0x7fcf2c1054a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.763+0000 7fcf30a31700 1 -- 172.21.15.78:0/4045207786 >> 172.21.15.78:0/4045207786 conn(0x7fcf2c0fa1e0 msgr2=0x7fcf2c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:58:39.770 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.764+0000 7fcf30a31700 1 -- 172.21.15.78:0/4045207786 shutdown_connections 2023-07-14T18:58:39.771 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.764+0000 7fcf30a31700 1 -- 172.21.15.78:0/4045207786 wait complete. 2023-07-14T18:58:39.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf30a31700 1 Processor -- start 2023-07-14T18:58:39.772 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf30a31700 1 -- start start 2023-07-14T18:58:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf30a31700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2c1007a0 0x7fcf2c195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:58:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf30a31700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf2c1030c0 0x7fcf2c195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:58:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf30a31700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1059e0 0x7fcf2c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:58:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf30a31700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c19d3f0 con 0x7fcf2c1059e0 2023-07-14T18:58:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf30a31700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c19d530 con 0x7fcf2c1030c0 2023-07-14T18:58:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf30a31700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf2c19d6c0 con 0x7fcf2c1007a0 2023-07-14T18:58:39.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1059e0 0x7fcf2c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:58:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1059e0 0x7fcf2c19ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33636/0 (socket says 172.21.15.78:33636) 2023-07-14T18:58:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2ad9d700 1 -- 172.21.15.78:0/2474560846 learned_addr learned my addr 172.21.15.78:0/2474560846 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:58:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2a59c700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2c1007a0 0x7fcf2c195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:58:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf29d9b700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf2c1030c0 0x7fcf2c195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:58:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2ad9d700 1 -- 172.21.15.78:0/2474560846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2c1007a0 msgr2=0x7fcf2c195860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2ad9d700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2c1007a0 0x7fcf2c195860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2ad9d700 1 -- 172.21.15.78:0/2474560846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf2c1030c0 msgr2=0x7fcf2c195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:39.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2ad9d700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf2c1030c0 0x7fcf2c195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf2ad9d700 1 -- 172.21.15.78:0/2474560846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf140095f0 con 0x7fcf2c1059e0 2023-07-14T18:58:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.765+0000 7fcf29d9b700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf2c1030c0 0x7fcf2c195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T18:58:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.766+0000 7fcf2ad9d700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1059e0 0x7fcf2c19ce20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcf24007ee0 tx=0x7fcf2400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:58:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.766+0000 7fcf237fe700 1 -- 172.21.15.78:0/2474560846 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24004db0 con 0x7fcf2c1059e0 2023-07-14T18:58:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.766+0000 7fcf237fe700 1 -- 172.21.15.78:0/2474560846 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcf2400f040 con 0x7fcf2c1059e0 2023-07-14T18:58:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.766+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf2c19d9a0 con 0x7fcf2c1059e0 2023-07-14T18:58:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.766+0000 7fcf237fe700 1 -- 172.21.15.78:0/2474560846 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf24005870 con 0x7fcf2c1059e0 2023-07-14T18:58:39.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.766+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf2c19dea0 con 0x7fcf2c1059e0 2023-07-14T18:58:39.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.767+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf2c0fbd50 con 0x7fcf2c1059e0 2023-07-14T18:58:39.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.768+0000 7fcf237fe700 1 -- 172.21.15.78:0/2474560846 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 15) v1 ==== 135659+0+0 (secure 0 0 0) 0x7fcf24005020 con 0x7fcf2c1059e0 2023-07-14T18:58:39.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.768+0000 7fcf237fe700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcf180a1140 0x7fcf180a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:58:39.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.768+0000 7fcf2a59c700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcf180a1140 0x7fcf180a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:58:39.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.768+0000 7fcf237fe700 1 -- 172.21.15.78:0/2474560846 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 2851+0+0 (secure 0 0 0) 0x7fcf240cabd0 con 0x7fcf2c1059e0 2023-07-14T18:58:39.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.769+0000 7fcf2a59c700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcf180a1140 0x7fcf180a35f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00ac50 tx=0x7fcf1c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:58:39.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.773+0000 7fcf237fe700 1 -- 172.21.15.78:0/2474560846 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcf2404e780 con 0x7fcf2c1059e0 2023-07-14T18:58:39.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:39 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T18:58:39.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:39 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T18:58:39.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:39.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:39 smithi102 ceph-mon[129280]: osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] boot 2023-07-14T18:58:39.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:39 smithi102 ceph-mon[129280]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:58:39.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:39.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:39 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T18:58:39.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:39 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T18:58:39.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:39.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:39 smithi088 ceph-mon[128973]: osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] boot 2023-07-14T18:58:39.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:39 smithi088 ceph-mon[128973]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:58:39.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:39.966+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf2c060a80 con 0x7fcf180a1140 2023-07-14T18:58:40.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:39 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T18:58:40.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:39 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T18:58:40.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:40.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:39 smithi078 ceph-mon[127425]: osd.2 [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] boot 2023-07-14T18:58:40.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:39 smithi078 ceph-mon[127425]: osdmap e18: 3 total, 3 up, 3 in 2023-07-14T18:58:40.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-07-14T18:58:40.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:40 smithi088 ceph-mon[128973]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:58:40.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:40 smithi088 ceph-mon[128973]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:58:40.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:40 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:58:40.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:40 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:58:40.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:40 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:40.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:40 smithi088 ceph-mon[128973]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:40.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:40 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:58:40.972 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:40 smithi078 ceph-mon[127425]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:58:40.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:40 smithi078 ceph-mon[127425]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:58:40.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:40 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:58:40.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:40 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:58:40.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:40 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:40.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:40 smithi078 ceph-mon[127425]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:40.973 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:40 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:58:41.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:40 smithi102 ceph-mon[129280]: osdmap e19: 3 total, 3 up, 3 in 2023-07-14T18:58:41.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:40 smithi102 ceph-mon[129280]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:58:41.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:40 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:58:41.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:40 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:58:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:40 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:40 smithi102 ceph-mon[129280]: pgmap v74: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:40 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:58:41.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:58:41.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:41 smithi078 ceph-mon[127425]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:58:41.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:58:41.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:58:41.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:41 smithi088 ceph-mon[128973]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:58:41.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:58:42.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-07-14T18:58:42.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:41 smithi102 ceph-mon[129280]: osdmap e20: 3 total, 3 up, 3 in 2023-07-14T18:58:42.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T18:58:42.265 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:58:41 smithi078 sudo[143627]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:58:42.265 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:58:41 smithi078 sudo[143627]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:42.266 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:58:41 smithi078 sudo[143627]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:42.266 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:58:42 smithi078 sudo[143627]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:42.549 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:58:42 smithi078 sudo[143650]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:58:42.549 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:58:42 smithi078 sudo[143650]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:42.549 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:58:42 smithi078 sudo[143650]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:42.549 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 18:58:42 smithi078 sudo[143650]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:42.788 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:42 smithi088 sudo[136289]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:58:42.788 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:42 smithi088 sudo[136289]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:42.788 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:42 smithi088 sudo[136289]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:42.789 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:42 smithi088 sudo[136289]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:42.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:58:42.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:42 smithi088 ceph-mon[128973]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:58:42.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:42 smithi088 ceph-mon[128973]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:42.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:42.789 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:42 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3154446052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e708d95-c299-4868-bbea-8ac0241d4118"}]: dispatch 2023-07-14T18:58:42.790 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:42 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3154446052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e708d95-c299-4868-bbea-8ac0241d4118"}]': finished 2023-07-14T18:58:42.790 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:42 smithi088 ceph-mon[128973]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:58:42.790 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:58:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:42 smithi078 ceph-mon[127425]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:58:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:42 smithi078 ceph-mon[127425]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:42 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3154446052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e708d95-c299-4868-bbea-8ac0241d4118"}]: dispatch 2023-07-14T18:58:43.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:42 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3154446052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e708d95-c299-4868-bbea-8ac0241d4118"}]': finished 2023-07-14T18:58:43.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:42 smithi078 ceph-mon[127425]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:58:43.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:43.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-07-14T18:58:43.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:42 smithi102 ceph-mon[129280]: osdmap e21: 3 total, 3 up, 3 in 2023-07-14T18:58:43.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:42 smithi102 ceph-mon[129280]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:43.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:43.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:42 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3154446052' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2e708d95-c299-4868-bbea-8ac0241d4118"}]: dispatch 2023-07-14T18:58:43.058 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:42 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3154446052' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2e708d95-c299-4868-bbea-8ac0241d4118"}]': finished 2023-07-14T18:58:43.059 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:42 smithi102 ceph-mon[129280]: osdmap e22: 4 total, 3 up, 4 in 2023-07-14T18:58:43.059 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:43.213 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:42 smithi088 sudo[136292]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:58:43.214 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:42 smithi088 sudo[136292]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:43.214 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:42 smithi088 sudo[136292]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:43.214 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 18:58:42 smithi088 sudo[136292]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:43.310 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:43 smithi102 sudo[137247]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:58:43.310 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:43 smithi102 sudo[137247]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:43.310 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:43 smithi102 sudo[137247]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:43.311 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:43 smithi102 sudo[137247]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:43.632 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:43 smithi102 sudo[137250]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:58:43.632 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:43 smithi102 sudo[137250]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:43.632 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:43 smithi102 sudo[137250]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:43.632 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 18:58:43 smithi102 sudo[137250]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:43.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:43 smithi078 sudo[143922]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:58:43.832 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:43 smithi078 sudo[143922]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:43.833 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:43 smithi078 sudo[143922]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:43.833 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:43 smithi078 sudo[143922]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:43.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:43.840+0000 7fcf237fe700 1 -- 172.21.15.78:0/2474560846 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcf24095d20 con 0x7fcf2c1059e0 2023-07-14T18:58:44.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:43 smithi078 sudo[144109]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:58:44.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:43 smithi078 sudo[144109]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:44.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:43 smithi078 sudo[144109]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:44.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:43 smithi078 sudo[144109]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:44.331 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 sudo[137253]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:58:44.332 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 sudo[137253]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:44.332 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 sudo[137253]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:44.332 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 sudo[137253]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:44.590 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:44.590 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/107787909' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:44.590 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:44.590 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 ceph-mon[128973]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:44.591 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 sudo[136295]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-07-14T18:58:44.591 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 sudo[136295]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:44.591 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 sudo[136295]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:44.591 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 sudo[136295]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 sudo[137256]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:58:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 sudo[137256]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 sudo[137256]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 sudo[137256]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:44.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/107787909' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:44.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:44.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:44 smithi102 ceph-mon[129280]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:44 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/107787909' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:58:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:44 smithi078 ceph-mon[127425]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:44.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 sudo[136298]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-07-14T18:58:44.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 sudo[136298]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-07-14T18:58:44.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 sudo[136298]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-07-14T18:58:44.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:44 smithi088 sudo[136298]: pam_unix(sudo:session): session closed for user root 2023-07-14T18:58:45.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:45.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:45.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:58:45.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:45.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:45.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:45.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:45 smithi102 ceph-mon[129280]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:45.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:45.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:45.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:58:45.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:45.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:45.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:45.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:45 smithi088 ceph-mon[128973]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:45.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:45.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:45.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: mgrmap e16: x(active, since 2m), standbys: z, y 2023-07-14T18:58:45.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:45.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:45.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-07-14T18:58:45.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-07-14T18:58:45.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-07-14T18:58:45.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-07-14T18:58:45.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:45 smithi078 ceph-mon[127425]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-07-14T18:58:46.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:46 smithi078 ceph-mon[127425]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:46.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:46 smithi102 ceph-mon[129280]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:46.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:46 smithi088 ceph-mon[128973]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:49.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:49 smithi102 ceph-mon[129280]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:49.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:49 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:58:49.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:49 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:49.458 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:49 smithi078 ceph-mon[127425]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:49.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:49 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:58:49.459 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:49 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:49.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:49 smithi088 ceph-mon[128973]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-07-14T18:58:49.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:49 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-07-14T18:58:49.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:49 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:50 smithi102 ceph-mon[129280]: Deploying daemon osd.3 on smithi078 2023-07-14T18:58:50.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:50 smithi088 ceph-mon[128973]: Deploying daemon osd.3 on smithi078 2023-07-14T18:58:50.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:50 smithi078 ceph-mon[127425]: Deploying daemon osd.3 on smithi078 2023-07-14T18:58:51.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:51 smithi078 ceph-mon[127425]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T18:58:51.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:51 smithi102 ceph-mon[129280]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T18:58:51.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:51 smithi088 ceph-mon[128973]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 44 KiB/s wr, 3 op/s 2023-07-14T18:58:53.271 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:53 smithi078 ceph-mon[127425]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T18:58:53.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:53 smithi102 ceph-mon[129280]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T18:58:53.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:53 smithi088 ceph-mon[128973]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 33 KiB/s wr, 2 op/s 2023-07-14T18:58:54.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:54.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:54.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:54 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:54.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:54 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:54.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:54 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:54.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:54 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:54.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:54 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:54.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:54 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:54.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:54 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:55.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:55 smithi102 ceph-mon[129280]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:58:55.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:55 smithi088 ceph-mon[128973]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:58:55.794 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:55 smithi078 ceph-mon[127425]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:58:56.274 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 3 on host 'smithi078' 2023-07-14T18:58:56.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.268+0000 7fcf237fe700 1 -- 172.21.15.78:0/2474560846 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fcf2c060a80 con 0x7fcf180a1140 2023-07-14T18:58:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcf180a1140 msgr2=0x7fcf180a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcf180a1140 0x7fcf180a35f0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcf1c00ac50 tx=0x7fcf1c00a380 comp rx=0 tx=0).stop 2023-07-14T18:58:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1059e0 msgr2=0x7fcf2c19ce20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:58:56.277 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1059e0 0x7fcf2c19ce20 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcf24007ee0 tx=0x7fcf2400d3b0 comp rx=0 tx=0).stop 2023-07-14T18:58:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 shutdown_connections 2023-07-14T18:58:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcf180a1140 0x7fcf180a35f0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcf2c1059e0 0x7fcf2c19ce20 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf2c1007a0 0x7fcf2c195860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:56.278 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 --2- 172.21.15.78:0/2474560846 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcf2c1030c0 0x7fcf2c195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:58:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 >> 172.21.15.78:0/2474560846 conn(0x7fcf2c0fa1e0 msgr2=0x7fcf2c101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:58:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 shutdown_connections 2023-07-14T18:58:56.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:58:56.271+0000 7fcf30a31700 1 -- 172.21.15.78:0/2474560846 wait complete. 2023-07-14T18:58:56.707 DEBUG:teuthology.orchestra.run.smithi078:osd.3> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.3.service 2023-07-14T18:58:56.711 INFO:tasks.cephadm:Deploying osd.4 on smithi088 with /dev/vg_nvme/lv_3... 2023-07-14T18:58:56.711 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:58:56.743 INFO:journalctl@ceph.osd.3.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T18:58:56.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:56.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:56.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T18:58:56.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:56.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:56.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:56.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T18:58:56.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:56.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:56.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:57.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:57.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:57.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:57.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:57.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:57.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2023-07-14T18:58:57.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:57.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:58:57.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:57.276 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:58:57.534 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 14 18:58:57 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-3[145928]: 2023-07-14T18:58:57.269+0000 7f0eebf37640 -1 osd.3 0 log_to_monitors true 2023-07-14T18:58:57.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:57 smithi078 ceph-mon[127425]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:58:57.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:57 smithi078 ceph-mon[127425]: from='osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:58:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:57 smithi088 ceph-mon[128973]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:58:57.820 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:57 smithi088 ceph-mon[128973]: from='osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:58:57.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:57 smithi102 ceph-mon[129280]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:58:57.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:57 smithi102 ceph-mon[129280]: from='osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-07-14T18:58:58.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:58 smithi088 ceph-mon[128973]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:58:58.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:58 smithi088 ceph-mon[128973]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:58:58.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:58 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:58.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:58 smithi088 ceph-mon[128973]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:58:58.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:58 smithi088 ceph-mon[128973]: from='osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:58:58.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:58 smithi088 ceph-mon[128973]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:58:58.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:58 smithi102 ceph-mon[129280]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:58:58.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:58 smithi102 ceph-mon[129280]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:58:58.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:58 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:58.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:58 smithi102 ceph-mon[129280]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:58:58.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:58 smithi102 ceph-mon[129280]: from='osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:58:58.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:58 smithi102 ceph-mon[129280]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:58:58.941 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:58:58.941 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:58:58.941 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-14T18:58:58.941 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-14T18:58:58.942 INFO:teuthology.orchestra.run.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0107954 s, 971 MB/s 2023-07-14T18:58:58.942 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-14T18:58:58.977 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:58 smithi078 ceph-mon[127425]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-07-14T18:58:58.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:58 smithi078 ceph-mon[127425]: osdmap e23: 4 total, 3 up, 4 in 2023-07-14T18:58:58.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:58 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:58.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:58 smithi078 ceph-mon[127425]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:58:58.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:58 smithi078 ceph-mon[127425]: from='osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T18:58:58.978 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:58 smithi078 ceph-mon[127425]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail; 921 B/s rd, 26 KiB/s wr, 2 op/s 2023-07-14T18:58:58.979 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 14 18:58:58 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-3[145928]: 2023-07-14T18:58:58.618+0000 7f0ee1db0700 -1 osd.3 0 waiting for initial osdmap 2023-07-14T18:58:58.979 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 14 18:58:58 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-3[145928]: 2023-07-14T18:58:58.623+0000 7f0edbba2700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:58:59.428 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_3 2023-07-14T18:58:59.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T18:58:59.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:58:59.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:59.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:59.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi078 2023-07-14T18:58:59.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:59.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:59.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:59.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:59.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi078 to 8584M 2023-07-14T18:58:59.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:59.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:59.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:59.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:58:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T18:58:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:58:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:59.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:58:59.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi078 2023-07-14T18:58:59.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:59.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:59.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:59.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:58:59.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi078 to 8584M 2023-07-14T18:58:59.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:59.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:58:59.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:58:59.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:58:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:58:59.977 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T18:59:00.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T18:59:00.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: osdmap e24: 4 total, 3 up, 4 in 2023-07-14T18:59:00.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:59:00.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:59:00.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi078 2023-07-14T18:59:00.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:00.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:00.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:00.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:00.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi078 to 8584M 2023-07-14T18:59:00.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:00.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:00.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:00.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:58:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:00.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:00 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T18:59:00.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:00 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T18:59:00.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:00 smithi102 ceph-mon[129280]: osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] boot 2023-07-14T18:59:00.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:00 smithi102 ceph-mon[129280]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:59:00.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:59:00.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:00 smithi102 ceph-mon[129280]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:00.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:00 smithi102 ceph-mon[129280]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:59:00.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:00 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T18:59:00.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:00 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T18:59:00.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:00 smithi088 ceph-mon[128973]: osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] boot 2023-07-14T18:59:00.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:00 smithi088 ceph-mon[128973]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:59:00.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:59:00.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:00 smithi088 ceph-mon[128973]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:00.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:00 smithi088 ceph-mon[128973]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:59:01.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:00 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T18:59:01.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:00 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T18:59:01.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:00 smithi078 ceph-mon[127425]: osd.3 [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] boot 2023-07-14T18:59:01.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:00 smithi078 ceph-mon[127425]: osdmap e25: 4 total, 4 up, 4 in 2023-07-14T18:59:01.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-07-14T18:59:01.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:00 smithi078 ceph-mon[127425]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:01.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:00 smithi078 ceph-mon[127425]: osdmap e26: 4 total, 4 up, 4 in 2023-07-14T18:59:01.216 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 -- 172.21.15.88:0/2213767290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 msgr2=0x7f1cf41005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 --2- 172.21.15.88:0/2213767290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 0x7f1cf41005d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009990 tx=0x7f1ce8009ca0 comp rx=0 tx=0).stop 2023-07-14T18:59:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 -- 172.21.15.88:0/2213767290 shutdown_connections 2023-07-14T18:59:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 --2- 172.21.15.88:0/2213767290 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 0x7f1cf41005d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 --2- 172.21.15.88:0/2213767290 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cf40ff870 0x7f1cf4107120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:01.217 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 --2- 172.21.15.88:0/2213767290 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf40feec0 0x7f1cf40ff330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 -- 172.21.15.88:0/2213767290 >> 172.21.15.88:0/2213767290 conn(0x7f1cf40f99c0 msgr2=0x7f1cf40fbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:59:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 -- 172.21.15.88:0/2213767290 shutdown_connections 2023-07-14T18:59:01.218 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.212+0000 7f1cfa101700 1 -- 172.21.15.88:0/2213767290 wait complete. 2023-07-14T18:59:01.219 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cfa101700 1 Processor -- start 2023-07-14T18:59:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cfa101700 1 -- start start 2023-07-14T18:59:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cfa101700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cf40feec0 0x7f1cf40fe910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cfa101700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf40ff870 0x7f1cf419a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cfa101700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 0x7f1cf419cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:01.220 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cfa101700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf419d420 con 0x7f1cf40ff870 2023-07-14T18:59:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cfa101700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf419d590 con 0x7f1cf41001c0 2023-07-14T18:59:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cfa101700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf419d750 con 0x7f1cf40feec0 2023-07-14T18:59:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 0x7f1cf419cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 0x7f1cf419cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:59898/0 (socket says 172.21.15.88:59898) 2023-07-14T18:59:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 -- 172.21.15.88:0/1914790107 learned_addr learned my addr 172.21.15.88:0/1914790107 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T18:59:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf2ffd700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf40ff870 0x7f1cf419a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:01.221 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf37fe700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cf40feec0 0x7f1cf40fe910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 -- 172.21.15.88:0/1914790107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cf40feec0 msgr2=0x7f1cf40fe910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cf40feec0 0x7f1cf40fe910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 -- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf40ff870 msgr2=0x7f1cf419a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf40ff870 0x7f1cf419a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 -- 172.21.15.88:0/1914790107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce80095f0 con 0x7f1cf41001c0 2023-07-14T18:59:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.213+0000 7f1cf3fff700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 0x7f1cf419cc80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1cec007d70 tx=0x7f1cec00a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:59:01.222 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.214+0000 7f1cf0ff9700 1 -- 172.21.15.88:0/1914790107 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec00ad40 con 0x7f1cf41001c0 2023-07-14T18:59:01.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.214+0000 7f1cf0ff9700 1 -- 172.21.15.88:0/1914790107 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1cec0128b0 con 0x7f1cf41001c0 2023-07-14T18:59:01.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.214+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf419da30 con 0x7f1cf41001c0 2023-07-14T18:59:01.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.214+0000 7f1cf0ff9700 1 -- 172.21.15.88:0/1914790107 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cec0116b0 con 0x7f1cf41001c0 2023-07-14T18:59:01.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.214+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf419dee0 con 0x7f1cf41001c0 2023-07-14T18:59:01.223 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.215+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf4105c10 con 0x7f1cf41001c0 2023-07-14T18:59:01.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.216+0000 7f1cf0ff9700 1 -- 172.21.15.88:0/1914790107 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1cec012a20 con 0x7f1cf41001c0 2023-07-14T18:59:01.224 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.216+0000 7f1cf0ff9700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1cdc0a1150 0x7f1cdc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:01.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.216+0000 7f1cf0ff9700 1 -- 172.21.15.88:0/1914790107 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 3698+0+0 (secure 0 0 0) 0x7f1cec0cba10 con 0x7f1cf41001c0 2023-07-14T18:59:01.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.217+0000 7f1cf37fe700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1cdc0a1150 0x7f1cdc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:01.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.217+0000 7f1cf37fe700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1cdc0a1150 0x7f1cdc0a3600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098e0 tx=0x7f1ce8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:59:01.225 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.221+0000 7f1cf0ff9700 1 -- 172.21.15.88:0/1914790107 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1cec099df0 con 0x7f1cf41001c0 2023-07-14T18:59:01.417 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:01.413+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1cf4060a80 con 0x7f1cdc0a1150 2023-07-14T18:59:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:01 smithi102 ceph-mon[129280]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:01 smithi102 ceph-mon[129280]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:59:01.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:01 smithi088 ceph-mon[128973]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:01.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:01.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:01.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:01.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:01 smithi088 ceph-mon[128973]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:59:02.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:01 smithi078 ceph-mon[127425]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:02.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:02.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:02.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:02.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:01 smithi078 ceph-mon[127425]: osdmap e27: 4 total, 4 up, 4 in 2023-07-14T18:59:02.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:02 smithi088 ceph-mon[128973]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:03.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:02 smithi078 ceph-mon[127425]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:03.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:02 smithi102 ceph-mon[129280]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:03.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:03 smithi088 ceph-mon[128973]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:59:04.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:03 smithi078 ceph-mon[127425]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:59:04.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:03 smithi102 ceph-mon[129280]: osdmap e28: 4 total, 4 up, 4 in 2023-07-14T18:59:04.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:04 smithi088 ceph-mon[128973]: osdmap e29: 4 total, 4 up, 4 in 2023-07-14T18:59:04.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:04 smithi088 ceph-mon[128973]: from='client.? 172.21.15.88:0/1827746266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]: dispatch 2023-07-14T18:59:04.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:04 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]: dispatch 2023-07-14T18:59:04.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:04 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]': finished 2023-07-14T18:59:04.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:04 smithi088 ceph-mon[128973]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:59:04.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:04 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:04.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:04 smithi088 ceph-mon[128973]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 164 KiB/s, 0 objects/s recovering 2023-07-14T18:59:04.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:04 smithi088 ceph-mon[128973]: from='client.? 172.21.15.88:0/2064835879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:05.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:04 smithi078 ceph-mon[127425]: osdmap e29: 4 total, 4 up, 4 in 2023-07-14T18:59:05.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:04 smithi078 ceph-mon[127425]: from='client.? 172.21.15.88:0/1827746266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]: dispatch 2023-07-14T18:59:05.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:04 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]: dispatch 2023-07-14T18:59:05.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:04 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]': finished 2023-07-14T18:59:05.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:04 smithi078 ceph-mon[127425]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:59:05.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:04 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:05.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:04 smithi078 ceph-mon[127425]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 164 KiB/s, 0 objects/s recovering 2023-07-14T18:59:05.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:04 smithi078 ceph-mon[127425]: from='client.? 172.21.15.88:0/2064835879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:04 smithi102 ceph-mon[129280]: osdmap e29: 4 total, 4 up, 4 in 2023-07-14T18:59:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:04 smithi102 ceph-mon[129280]: from='client.? 172.21.15.88:0/1827746266' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]: dispatch 2023-07-14T18:59:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:04 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]: dispatch 2023-07-14T18:59:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:04 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f2a8821e-94f2-4927-a1c6-c1bcce365fea"}]': finished 2023-07-14T18:59:05.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:04 smithi102 ceph-mon[129280]: osdmap e30: 5 total, 4 up, 5 in 2023-07-14T18:59:05.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:04 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:05.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:04 smithi102 ceph-mon[129280]: pgmap v97: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 164 KiB/s, 0 objects/s recovering 2023-07-14T18:59:05.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:04 smithi102 ceph-mon[129280]: from='client.? 172.21.15.88:0/2064835879' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:06.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:06 smithi102 ceph-mon[129280]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2023-07-14T18:59:06.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:06 smithi088 ceph-mon[128973]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2023-07-14T18:59:07.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:06 smithi078 ceph-mon[127425]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 128 KiB/s, 0 objects/s recovering 2023-07-14T18:59:09.380 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:09 smithi088 ceph-mon[128973]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:59:09.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:09 smithi102 ceph-mon[129280]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:59:09.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:09 smithi078 ceph-mon[127425]: pgmap v99: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:59:11.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:11 smithi102 ceph-mon[129280]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:59:11.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:11 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:59:11.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:11 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:11.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:11 smithi102 ceph-mon[129280]: Deploying daemon osd.4 on smithi088 2023-07-14T18:59:11.391 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:11 smithi088 ceph-mon[128973]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:59:11.392 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:11 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:59:11.392 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:11 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:11.392 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:11 smithi088 ceph-mon[128973]: Deploying daemon osd.4 on smithi088 2023-07-14T18:59:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:11 smithi078 ceph-mon[127425]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T18:59:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:11 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-07-14T18:59:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:11 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:11 smithi078 ceph-mon[127425]: Deploying daemon osd.4 on smithi088 2023-07-14T18:59:13.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:13 smithi088 ceph-mon[128973]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:13.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:13 smithi078 ceph-mon[127425]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:13.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:13 smithi102 ceph-mon[129280]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:15.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:15 smithi088 ceph-mon[128973]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:15.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:15 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:15.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:15 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:15.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:15 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:15.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:15 smithi078 ceph-mon[127425]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:15.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:15 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:15.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:15 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:15.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:15 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:15.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:15 smithi102 ceph-mon[129280]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:15.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:15 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:15.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:15 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:15.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:15 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:16.765 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:16 smithi088 ceph-mon[128973]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:16.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:16 smithi102 ceph-mon[129280]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:17.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:16 smithi078 ceph-mon[127425]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:17.777 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 4 on host 'smithi088' 2023-07-14T18:59:17.778 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.773+0000 7f1cf0ff9700 1 -- 172.21.15.88:0/1914790107 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1cf4060a80 con 0x7f1cdc0a1150 2023-07-14T18:59:17.780 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1cdc0a1150 msgr2=0x7f1cdc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:17.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1cdc0a1150 0x7f1cdc0a3600 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1ce80098e0 tx=0x7f1ce8000bc0 comp rx=0 tx=0).stop 2023-07-14T18:59:17.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 msgr2=0x7f1cf419cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:17.781 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 0x7f1cf419cc80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f1cec007d70 tx=0x7f1cec00a340 comp rx=0 tx=0).stop 2023-07-14T18:59:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 shutdown_connections 2023-07-14T18:59:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1cdc0a1150 0x7f1cdc0a3600 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1cf41001c0 0x7f1cf419cc80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1cf40feec0 0x7f1cf40fe910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 --2- 172.21.15.88:0/1914790107 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1cf40ff870 0x7f1cf419a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 >> 172.21.15.88:0/1914790107 conn(0x7f1cf40f99c0 msgr2=0x7f1cf40fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:59:17.782 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 shutdown_connections 2023-07-14T18:59:17.783 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T18:59:17.775+0000 7f1cfa101700 1 -- 172.21.15.88:0/1914790107 wait complete. 2023-07-14T18:59:18.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.261 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:18.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:18.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.262 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:18.263 DEBUG:teuthology.orchestra.run.smithi088:osd.4> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.4.service 2023-07-14T18:59:18.265 INFO:tasks.cephadm:Deploying osd.5 on smithi102 with /dev/vg_nvme/lv_3... 2023-07-14T18:59:18.266 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-07-14T18:59:18.294 INFO:journalctl@ceph.osd.4.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T18:59:18.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:18.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:18.298 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.299 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.299 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.299 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:18.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:18.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:18.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:18.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:18.838 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:59:18.963 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 18:59:18 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-4[139487]: 2023-07-14T18:59:18.704+0000 7f304866d640 -1 osd.4 0 log_to_monitors true 2023-07-14T18:59:19.255 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:18 smithi102 ceph-mon[129280]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:19.255 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:18 smithi102 ceph-mon[129280]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:59:19.255 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:18 smithi102 ceph-mon[129280]: from='osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:59:19.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:18 smithi078 ceph-mon[127425]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:19.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:18 smithi078 ceph-mon[127425]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:59:19.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:18 smithi078 ceph-mon[127425]: from='osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:59:19.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:19 smithi088 ceph-mon[128973]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:19.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:19 smithi088 ceph-mon[128973]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:59:19.325 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:19 smithi088 ceph-mon[128973]: from='osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-07-14T18:59:20.291 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-07-14T18:59:20.291 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-07-14T18:59:20.291 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2023-07-14T18:59:20.292 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2023-07-14T18:59:20.292 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111432 s, 941 MB/s 2023-07-14T18:59:20.292 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2023-07-14T18:59:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:20 smithi078 ceph-mon[127425]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:59:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:20 smithi078 ceph-mon[127425]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:59:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:20 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:20 smithi078 ceph-mon[127425]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:59:20.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:20 smithi078 ceph-mon[127425]: from='osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:59:20.300 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:20 smithi088 ceph-mon[128973]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:59:20.300 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:20 smithi088 ceph-mon[128973]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:59:20.300 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:20 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:20.300 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:20 smithi088 ceph-mon[128973]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:59:20.300 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:20 smithi088 ceph-mon[128973]: from='osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:59:20.301 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 18:59:20 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-4[139487]: 2023-07-14T18:59:20.006+0000 7f303dce5700 -1 osd.4 0 waiting for initial osdmap 2023-07-14T18:59:20.301 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 18:59:20 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-4[139487]: 2023-07-14T18:59:20.011+0000 7f3039adb700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:59:20.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:20 smithi102 ceph-mon[129280]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-07-14T18:59:20.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:20 smithi102 ceph-mon[129280]: osdmap e31: 5 total, 4 up, 5 in 2023-07-14T18:59:20.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:20 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:20.327 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:20 smithi102 ceph-mon[129280]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:59:20.328 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:20 smithi102 ceph-mon[129280]: from='osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T18:59:20.690 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi102:vg_nvme/lv_3 2023-07-14T18:59:21.279 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T18:59:21.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T18:59:21.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: osdmap e32: 5 total, 4 up, 5 in 2023-07-14T18:59:21.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:21.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:21.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:21.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:21.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:21.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:21.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:21.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T18:59:21.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: osdmap e32: 5 total, 4 up, 5 in 2023-07-14T18:59:21.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:21.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:21.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:21.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:21.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:21.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:21.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:21.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T18:59:21.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: osdmap e32: 5 total, 4 up, 5 in 2023-07-14T18:59:21.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:21.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:21.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-07-14T18:59:21.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:21.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:21.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:21.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:21.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:21.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:22.044 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:22 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T18:59:22.044 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:22 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T18:59:22.045 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:22 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi088 2023-07-14T18:59:22.045 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:22 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-14T18:59:22.045 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:22 smithi102 ceph-mon[129280]: osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] boot 2023-07-14T18:59:22.045 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:22 smithi102 ceph-mon[129280]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:59:22.045 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:22 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:22 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T18:59:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:22 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T18:59:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:22 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi088 2023-07-14T18:59:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:22 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-14T18:59:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:22 smithi078 ceph-mon[127425]: osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] boot 2023-07-14T18:59:22.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:22 smithi078 ceph-mon[127425]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:59:22.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:22 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 -- 172.21.15.102:0/3956841563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 msgr2=0x7fe5b0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3956841563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 0x7fe5b0102b80 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe5a0009990 tx=0x7fe5a0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:59:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 -- 172.21.15.102:0/3956841563 shutdown_connections 2023-07-14T18:59:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3956841563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5b01059e0 0x7fe5b0107e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:22.448 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3956841563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 0x7fe5b0102b80 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3956841563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5b01030c0 0x7fe5b01054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 -- 172.21.15.102:0/3956841563 >> 172.21.15.102:0/3956841563 conn(0x7fe5b00fa260 msgr2=0x7fe5b00fc690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:59:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 -- 172.21.15.102:0/3956841563 shutdown_connections 2023-07-14T18:59:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.442+0000 7fe5b604c700 1 -- 172.21.15.102:0/3956841563 wait complete. 2023-07-14T18:59:22.449 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5b604c700 1 Processor -- start 2023-07-14T18:59:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5b604c700 1 -- start start 2023-07-14T18:59:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5b604c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 0x7fe5b01958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:22.450 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5b604c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5b01030c0 0x7fe5b0195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5b604c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5b01059e0 0x7fe5b019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5b604c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b019d470 con 0x7fe5b01059e0 2023-07-14T18:59:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5b604c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b019d5b0 con 0x7fe5b01030c0 2023-07-14T18:59:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5b604c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5b019d740 con 0x7fe5b01007a0 2023-07-14T18:59:22.451 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5af7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 0x7fe5b01958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:22.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5af7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 0x7fe5b01958e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52138/0 (socket says 172.21.15.102:52138) 2023-07-14T18:59:22.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5af7fe700 1 -- 172.21.15.102:0/3760746450 learned_addr learned my addr 172.21.15.102:0/3760746450 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T18:59:22.452 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5aeffd700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5b01030c0 0x7fe5b0195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:22.453 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5affff700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5b01059e0 0x7fe5b019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.443+0000 7fe5af7fe700 1 -- 172.21.15.102:0/3760746450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5b01030c0 msgr2=0x7fe5b0195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5af7fe700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5b01030c0 0x7fe5b0195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:22.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5af7fe700 1 -- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5b01059e0 msgr2=0x7fe5b019cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:22.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5af7fe700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5b01059e0 0x7fe5b019cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:22.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5af7fe700 1 -- 172.21.15.102:0/3760746450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5a00095f0 con 0x7fe5b01007a0 2023-07-14T18:59:22.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5affff700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5b01059e0 0x7fe5b019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T18:59:22.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5af7fe700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 0x7fe5b01958e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00059d0 tx=0x7fe5a0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:59:22.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5acff9700 1 -- 172.21.15.102:0/3760746450 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a001d070 con 0x7fe5b01007a0 2023-07-14T18:59:22.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5b019d9c0 con 0x7fe5b01007a0 2023-07-14T18:59:22.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5acff9700 1 -- 172.21.15.102:0/3760746450 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe5a0004080 con 0x7fe5b01007a0 2023-07-14T18:59:22.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5acff9700 1 -- 172.21.15.102:0/3760746450 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5a0022c50 con 0x7fe5b01007a0 2023-07-14T18:59:22.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.444+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5b019de30 con 0x7fe5b01007a0 2023-07-14T18:59:22.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.445+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5b00fbdd0 con 0x7fe5b01007a0 2023-07-14T18:59:22.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.446+0000 7fe5acff9700 1 -- 172.21.15.102:0/3760746450 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe5a0022470 con 0x7fe5b01007a0 2023-07-14T18:59:22.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.446+0000 7fe5acff9700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5980a1150 0x7fe5980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:22.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.446+0000 7fe5acff9700 1 -- 172.21.15.102:0/3760746450 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7fe5a00ccb50 con 0x7fe5b01007a0 2023-07-14T18:59:22.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.447+0000 7fe5aeffd700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5980a1150 0x7fe5980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:22.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.447+0000 7fe5aeffd700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5980a1150 0x7fe5980a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4005950 tx=0x7fe5a400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:59:22.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.451+0000 7fe5acff9700 1 -- 172.21.15.102:0/3760746450 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe5a005c780 con 0x7fe5b01007a0 2023-07-14T18:59:22.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:22 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T18:59:22.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:22 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T18:59:22.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:22 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi088 2023-07-14T18:59:22.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:22 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi088 to 8584M 2023-07-14T18:59:22.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:22 smithi088 ceph-mon[128973]: osd.4 [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] boot 2023-07-14T18:59:22.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:22 smithi088 ceph-mon[128973]: osdmap e33: 5 total, 5 up, 5 in 2023-07-14T18:59:22.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:22 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-07-14T18:59:22.652 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:22.647+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe5b0060a80 con 0x7fe5980a1150 2023-07-14T18:59:23.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:23 smithi078 ceph-mon[127425]: osdmap e34: 5 total, 5 up, 5 in 2023-07-14T18:59:23.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:23 smithi078 ceph-mon[127425]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:23.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:23.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:23.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:23 smithi102 ceph-mon[129280]: osdmap e34: 5 total, 5 up, 5 in 2023-07-14T18:59:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:23 smithi102 ceph-mon[129280]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:23.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:23 smithi088 ceph-mon[128973]: osdmap e34: 5 total, 5 up, 5 in 2023-07-14T18:59:23.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:23 smithi088 ceph-mon[128973]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:23.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:23.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:23.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:24 smithi078 ceph-mon[127425]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:24.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:24 smithi102 ceph-mon[129280]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:24.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:24 smithi088 ceph-mon[128973]: from='client.24217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:25.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:25 smithi078 ceph-mon[127425]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:25 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/27460365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]: dispatch 2023-07-14T18:59:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:25 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]: dispatch 2023-07-14T18:59:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:25 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]': finished 2023-07-14T18:59:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:25 smithi078 ceph-mon[127425]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:59:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:25.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:25 smithi102 ceph-mon[129280]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:25.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:25 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/27460365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]: dispatch 2023-07-14T18:59:25.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:25 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]: dispatch 2023-07-14T18:59:25.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:25 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]': finished 2023-07-14T18:59:25.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:25 smithi102 ceph-mon[129280]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:59:25.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:25.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:25 smithi088 ceph-mon[128973]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:25 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/27460365' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]: dispatch 2023-07-14T18:59:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:25 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]: dispatch 2023-07-14T18:59:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:25 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7abc51f7-2397-4f2a-8046-0b93c4a56115"}]': finished 2023-07-14T18:59:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:25 smithi088 ceph-mon[128973]: osdmap e35: 6 total, 5 up, 6 in 2023-07-14T18:59:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:26.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:26 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/365009727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:26.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:26 smithi102 ceph-mon[129280]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:26.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:26 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/365009727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:26.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:26 smithi088 ceph-mon[128973]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:27.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:26 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/365009727' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:27.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:26 smithi078 ceph-mon[127425]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:29.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:29 smithi102 ceph-mon[129280]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:29.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:29 smithi088 ceph-mon[128973]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:29.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:29 smithi078 ceph-mon[127425]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:31.353 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:31 smithi102 ceph-mon[129280]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:31 smithi088 ceph-mon[128973]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:31.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:31 smithi078 ceph-mon[127425]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:32.215 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:32 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:59:32.216 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:32 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:32.216 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:32 smithi102 ceph-mon[129280]: Deploying daemon osd.5 on smithi102 2023-07-14T18:59:32.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:32 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:59:32.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:32 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:32.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:32 smithi088 ceph-mon[128973]: Deploying daemon osd.5 on smithi102 2023-07-14T18:59:32.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:32 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-07-14T18:59:32.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:32 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:32.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:32 smithi078 ceph-mon[127425]: Deploying daemon osd.5 on smithi102 2023-07-14T18:59:33.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:33 smithi102 ceph-mon[129280]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:33.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:33 smithi088 ceph-mon[128973]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:33.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:33 smithi078 ceph-mon[127425]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:35.454 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:35 smithi102 ceph-mon[129280]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:35.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:35 smithi088 ceph-mon[128973]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:35.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:35 smithi078 ceph-mon[127425]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:36.957 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:36 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:36.957 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:36 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:36.957 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:36 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:36.957 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:36 smithi102 ceph-mon[129280]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:37.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:36 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:37.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:36 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:37.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:36 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:37.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:36 smithi078 ceph-mon[127425]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:37.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:36 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:37.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:36 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:37.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:36 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:37.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:36 smithi088 ceph-mon[128973]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:38.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.455+0000 7fe5acff9700 1 -- 172.21.15.102:0/3760746450 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe5b0060a80 con 0x7fe5980a1150 2023-07-14T18:59:38.460 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 5 on host 'smithi102' 2023-07-14T18:59:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5980a1150 msgr2=0x7fe5980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:38.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5980a1150 0x7fe5980a3600 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe5a4005950 tx=0x7fe5a400b410 comp rx=0 tx=0).stop 2023-07-14T18:59:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 msgr2=0x7fe5b01958e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 0x7fe5b01958e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fe5a00059d0 tx=0x7fe5a0004730 comp rx=0 tx=0).stop 2023-07-14T18:59:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 shutdown_connections 2023-07-14T18:59:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5980a1150 0x7fe5980a3600 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5b01059e0 0x7fe5b019cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5b01007a0 0x7fe5b01958e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:38.463 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 --2- 172.21.15.102:0/3760746450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5b01030c0 0x7fe5b0195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:38.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 >> 172.21.15.102:0/3760746450 conn(0x7fe5b00fa260 msgr2=0x7fe5b0101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:59:38.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 shutdown_connections 2023-07-14T18:59:38.464 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T18:59:38.457+0000 7fe5b604c700 1 -- 172.21.15.102:0/3760746450 wait complete. 2023-07-14T18:59:38.871 DEBUG:teuthology.orchestra.run.smithi102:osd.5> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.5.service 2023-07-14T18:59:38.873 INFO:tasks.cephadm:Deploying osd.6 on smithi078 with /dev/vg_nvme/lv_2... 2023-07-14T18:59:38.874 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T18:59:38.904 INFO:journalctl@ceph.osd.5.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T18:59:39.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:39.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:39.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:39.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:39.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:39.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:39.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:39.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:39.442 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:59:39.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:39.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:39.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:39.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:39.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:39.882 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 18:59:39 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T18:59:39.740+0000 7fb30db2d640 -1 osd.5 0 log_to_monitors true 2023-07-14T18:59:40.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:40 smithi078 ceph-mon[127425]: from='osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:59:40.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:40 smithi078 ceph-mon[127425]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:59:40.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:40 smithi102 ceph-mon[129280]: from='osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:59:40.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:40 smithi102 ceph-mon[129280]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:59:40.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:40 smithi088 ceph-mon[128973]: from='osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:59:40.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:40 smithi088 ceph-mon[128973]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-07-14T18:59:40.977 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T18:59:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T18:59:40.978 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-07-14T18:59:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-07-14T18:59:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130902 s, 801 MB/s 2023-07-14T18:59:40.978 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-07-14T18:59:41.131 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T18:59:41.056+0000 7fb3031a5700 -1 osd.5 0 waiting for initial osdmap 2023-07-14T18:59:41.132 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T18:59:41.062+0000 7fb2fcf97700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T18:59:41.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:59:41.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:59:41.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:41.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:59:41.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:59:41.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:41.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:41.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T18:59:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: osdmap e37: 6 total, 5 up, 6 in 2023-07-14T18:59:41.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:41 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:41.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:59:41.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:59:41.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:41.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:59:41.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:59:41.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:41.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:41.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:41.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:41.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:41.301 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T18:59:41.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: osdmap e37: 6 total, 5 up, 6 in 2023-07-14T18:59:41.302 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:41 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:41.400 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi078:vg_nvme/lv_2 2023-07-14T18:59:41.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-07-14T18:59:41.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: osdmap e36: 6 total, 5 up, 6 in 2023-07-14T18:59:41.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:41.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:59:41.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T18:59:41.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-07-14T18:59:41.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:41.466 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T18:59:41.466 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.466 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:41.466 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T18:59:41.466 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:41.467 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T18:59:41.467 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: osdmap e37: 6 total, 5 up, 6 in 2023-07-14T18:59:41.467 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:41 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T18:59:42.367 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:42 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi102 2023-07-14T18:59:42.368 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:42 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi102 to 8584M 2023-07-14T18:59:42.368 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:42.368 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:42 smithi078 ceph-mon[127425]: osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] boot 2023-07-14T18:59:42.368 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:42 smithi078 ceph-mon[127425]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:59:42.368 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:42.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:42 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi102 2023-07-14T18:59:42.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:42 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi102 to 8584M 2023-07-14T18:59:42.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:42.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:42 smithi102 ceph-mon[129280]: osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] boot 2023-07-14T18:59:42.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:42 smithi102 ceph-mon[129280]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:59:42.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:42.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:42 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi102 2023-07-14T18:59:42.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:42 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi102 to 8584M 2023-07-14T18:59:42.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:42.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:42 smithi088 ceph-mon[128973]: osd.5 [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] boot 2023-07-14T18:59:42.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:42 smithi088 ceph-mon[128973]: osdmap e38: 6 total, 6 up, 6 in 2023-07-14T18:59:42.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-07-14T18:59:43.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 -- 172.21.15.78:0/88093272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 msgr2=0x7f12b41002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:43.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 --2- 172.21.15.78:0/88093272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 0x7f12b41002d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f12b0009990 tx=0x7f12b0009ca0 comp rx=0 tx=0).stop 2023-07-14T18:59:43.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 -- 172.21.15.78:0/88093272 shutdown_connections 2023-07-14T18:59:43.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 --2- 172.21.15.78:0/88093272 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12b40fec80 0x7f12b40ff090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:43.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 --2- 172.21.15.78:0/88093272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12b4100810 0x7f12b41070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 --2- 172.21.15.78:0/88093272 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 0x7f12b41002d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 -- 172.21.15.78:0/88093272 >> 172.21.15.78:0/88093272 conn(0x7f12b40fa2d0 msgr2=0x7f12b40fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:59:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 -- 172.21.15.78:0/88093272 shutdown_connections 2023-07-14T18:59:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.211+0000 7f12bc40d700 1 -- 172.21.15.78:0/88093272 wait complete. 2023-07-14T18:59:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12bc40d700 1 Processor -- start 2023-07-14T18:59:43.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12bc40d700 1 -- start start 2023-07-14T18:59:43.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12bc40d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12b40fec80 0x7f12b41947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:43.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12bc40d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 0x7f12b4194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:43.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12bc40d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12b4100810 0x7f12b419bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12bc40d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b419c380 con 0x7f12b40ffe80 2023-07-14T18:59:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12bc40d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b419c4c0 con 0x7f12b40fec80 2023-07-14T18:59:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12bc40d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b419c650 con 0x7f12b4100810 2023-07-14T18:59:43.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12ba1a9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12b40fec80 0x7f12b41947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:43.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12ba1a9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12b40fec80 0x7f12b41947f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.78:57192/0 (socket says 172.21.15.78:57192) 2023-07-14T18:59:43.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12ba1a9700 1 -- 172.21.15.78:0/2927307263 learned_addr learned my addr 172.21.15.78:0/2927307263 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T18:59:43.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12ba9aa700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12b4100810 0x7f12b419bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:43.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.212+0000 7f12b99a8700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 0x7f12b4194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:43.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ba1a9700 1 -- 172.21.15.78:0/2927307263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12b4100810 msgr2=0x7f12b419bdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:43.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ba1a9700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12b4100810 0x7f12b419bdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:43.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ba1a9700 1 -- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 msgr2=0x7f12b4194d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:43.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ba1a9700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 0x7f12b4194d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:43.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ba1a9700 1 -- 172.21.15.78:0/2927307263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b00095f0 con 0x7f12b40fec80 2023-07-14T18:59:43.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12b99a8700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 0x7f12b4194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T18:59:43.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ba1a9700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12b40fec80 0x7f12b41947f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f12a400d980 tx=0x7f12a400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:59:43.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ab7fe700 1 -- 172.21.15.78:0/2927307263 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4004020 con 0x7f12b40fec80 2023-07-14T18:59:43.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b419c930 con 0x7f12b40fec80 2023-07-14T18:59:43.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ab7fe700 1 -- 172.21.15.78:0/2927307263 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f12a400f690 con 0x7f12b40fec80 2023-07-14T18:59:43.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12ab7fe700 1 -- 172.21.15.78:0/2927307263 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4003a00 con 0x7f12b40fec80 2023-07-14T18:59:43.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.213+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b419ce30 con 0x7f12b40fec80 2023-07-14T18:59:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.214+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b410a2f0 con 0x7f12b40fec80 2023-07-14T18:59:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.215+0000 7f12ab7fe700 1 -- 172.21.15.78:0/2927307263 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f12a400f800 con 0x7f12b40fec80 2023-07-14T18:59:43.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.216+0000 7f12ab7fe700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12a00a11d0 0x7f12a00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T18:59:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.216+0000 7f12b99a8700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12a00a11d0 0x7f12a00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T18:59:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.216+0000 7f12ab7fe700 1 -- 172.21.15.78:0/2927307263 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f12a40cb9e0 con 0x7f12b40fec80 2023-07-14T18:59:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.216+0000 7f12b99a8700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12a00a11d0 0x7f12a00a3680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f12b000b5c0 tx=0x7f12b0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T18:59:43.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.220+0000 7f12ab7fe700 1 -- 172.21.15.78:0/2927307263 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f12a4095f30 con 0x7f12b40fec80 2023-07-14T18:59:43.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:43 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T18:59:43.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:43 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T18:59:43.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:43 smithi078 ceph-mon[127425]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:43.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:43 smithi078 ceph-mon[127425]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:59:43.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:43 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T18:59:43.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:43 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T18:59:43.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:43 smithi102 ceph-mon[129280]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:43.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:43 smithi102 ceph-mon[129280]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:59:43.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:43.414+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f12b4060a80 con 0x7f12a00a11d0 2023-07-14T18:59:43.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:43 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T18:59:43.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:43 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T18:59:43.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:43 smithi088 ceph-mon[128973]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:43.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:43 smithi088 ceph-mon[128973]: osdmap e39: 6 total, 6 up, 6 in 2023-07-14T18:59:44.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:44 smithi078 ceph-mon[127425]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:44.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:44.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:44.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:44.334 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:44 smithi078 ceph-mon[127425]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:59:44.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:44 smithi102 ceph-mon[129280]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:44.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:44.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:44.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:44.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:44 smithi102 ceph-mon[129280]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:59:44.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:44 smithi088 ceph-mon[128973]: from='client.24281 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T18:59:44.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T18:59:44.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T18:59:44.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:44.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:44 smithi088 ceph-mon[128973]: osdmap e40: 6 total, 6 up, 6 in 2023-07-14T18:59:45.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:45 smithi102 ceph-mon[129280]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:45.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:45 smithi088 ceph-mon[128973]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:45.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:45 smithi078 ceph-mon[127425]: pgmap v127: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:46 smithi102 ceph-mon[129280]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:59:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:46 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/4204622810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c02e47de-7c3a-41c9-bb4c-d339519b6a06"}]: dispatch 2023-07-14T18:59:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:46 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/4204622810' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c02e47de-7c3a-41c9-bb4c-d339519b6a06"}]': finished 2023-07-14T18:59:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:46 smithi102 ceph-mon[129280]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:59:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:59:46.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:46 smithi088 ceph-mon[128973]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:59:46.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:46 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/4204622810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c02e47de-7c3a-41c9-bb4c-d339519b6a06"}]: dispatch 2023-07-14T18:59:46.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:46 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/4204622810' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c02e47de-7c3a-41c9-bb4c-d339519b6a06"}]': finished 2023-07-14T18:59:46.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:46 smithi088 ceph-mon[128973]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:59:46.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:59:46.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:46 smithi078 ceph-mon[127425]: osdmap e41: 6 total, 6 up, 6 in 2023-07-14T18:59:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:46 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/4204622810' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c02e47de-7c3a-41c9-bb4c-d339519b6a06"}]: dispatch 2023-07-14T18:59:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:46 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/4204622810' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c02e47de-7c3a-41c9-bb4c-d339519b6a06"}]': finished 2023-07-14T18:59:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:46 smithi078 ceph-mon[127425]: osdmap e42: 7 total, 6 up, 7 in 2023-07-14T18:59:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T18:59:47.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:47 smithi102 ceph-mon[129280]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:47.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:47 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/56274427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:47.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:47 smithi088 ceph-mon[128973]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:47.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:47 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/56274427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:47.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:47 smithi078 ceph-mon[127425]: pgmap v130: 1 pgs: 1 remapped+peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:47.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:47 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/56274427' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T18:59:49.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:49 smithi102 ceph-mon[129280]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:49.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:49 smithi088 ceph-mon[128973]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:49.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:49 smithi078 ceph-mon[127425]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 163 MiB used, 536 GiB / 536 GiB avail 2023-07-14T18:59:51.358 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:51 smithi078 ceph-mon[127425]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:59:51.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:51 smithi102 ceph-mon[129280]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:59:51.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:51 smithi088 ceph-mon[128973]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T18:59:53.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:53 smithi102 ceph-mon[129280]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:59:53.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:53 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:59:53.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:53 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:53.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:53 smithi078 ceph-mon[127425]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:59:53.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:53 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:59:53.460 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:53 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:53 smithi088 ceph-mon[128973]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T18:59:53.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:53 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-07-14T18:59:53.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:53 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T18:59:54.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:54 smithi078 ceph-mon[127425]: Deploying daemon osd.6 on smithi078 2023-07-14T18:59:54.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:54 smithi102 ceph-mon[129280]: Deploying daemon osd.6 on smithi078 2023-07-14T18:59:54.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:54 smithi088 ceph-mon[128973]: Deploying daemon osd.6 on smithi078 2023-07-14T18:59:55.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:55 smithi102 ceph-mon[129280]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T18:59:55.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:55 smithi088 ceph-mon[128973]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T18:59:55.529 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:55 smithi078 ceph-mon[127425]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 64 KiB/s, 0 objects/s recovering 2023-07-14T18:59:56.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:56 smithi102 ceph-mon[129280]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T18:59:56.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:56 smithi088 ceph-mon[128973]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T18:59:57.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:56 smithi078 ceph-mon[127425]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T18:59:58.001 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:57 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:58.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:57 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:58.002 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:57 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:58.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:57 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:58.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:57 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:58.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:57 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:58.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:57 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:58.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:57 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T18:59:58.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:57 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T18:59:59.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 18:59:58 smithi078 ceph-mon[127425]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:59:59.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 18:59:58 smithi102 ceph-mon[129280]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:59:59.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 18:59:58 smithi088 ceph-mon[128973]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T18:59:59.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.720+0000 7f12ab7fe700 1 -- 172.21.15.78:0/2927307263 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f12b4060a80 con 0x7f12a00a11d0 2023-07-14T18:59:59.726 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 6 on host 'smithi078' 2023-07-14T18:59:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12a00a11d0 msgr2=0x7f12a00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12a00a11d0 0x7f12a00a3680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f12b000b5c0 tx=0x7f12b0005fb0 comp rx=0 tx=0).stop 2023-07-14T18:59:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12b40fec80 msgr2=0x7f12b41947f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T18:59:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12b40fec80 0x7f12b41947f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f12a400d980 tx=0x7f12a400dc90 comp rx=0 tx=0).stop 2023-07-14T18:59:59.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 shutdown_connections 2023-07-14T18:59:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12a00a11d0 0x7f12a00a3680 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12b40fec80 0x7f12b41947f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12b4100810 0x7f12b419bdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 --2- 172.21.15.78:0/2927307263 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12b40ffe80 0x7f12b4194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T18:59:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 >> 172.21.15.78:0/2927307263 conn(0x7f12b40fa2d0 msgr2=0x7f12b40fc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T18:59:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 shutdown_connections 2023-07-14T18:59:59.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T18:59:59.723+0000 7f12bc40d700 1 -- 172.21.15.78:0/2927307263 wait complete. 2023-07-14T19:00:00.150 DEBUG:teuthology.orchestra.run.smithi078:osd.6> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.6.service 2023-07-14T19:00:00.152 INFO:tasks.cephadm:Deploying osd.7 on smithi088 with /dev/vg_nvme/lv_2... 2023-07-14T19:00:00.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:00:00.183 INFO:journalctl@ceph.osd.6.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T19:00:00.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:00.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:00.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:00.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:00 smithi088 ceph-mon[128973]: overall HEALTH_OK 2023-07-14T19:00:00.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:00.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:00.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:00.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-mon[127425]: overall HEALTH_OK 2023-07-14T19:00:00.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:00.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:00.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:00.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:00.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:00 smithi102 ceph-mon[129280]: overall HEALTH_OK 2023-07-14T19:00:00.731 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:00:00.942 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:00:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:00:00.825+0000 7f3c89386640 -1 osd.6 0 log_to_monitors true 2023-07-14T19:00:01.609 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:01 smithi088 ceph-mon[128973]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:00:01.610 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:01 smithi088 ceph-mon[128973]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:00:01.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:01 smithi102 ceph-mon[129280]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:00:01.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:01 smithi102 ceph-mon[129280]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:00:01.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:01 smithi078 ceph-mon[127425]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:00:01.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:01 smithi078 ceph-mon[127425]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-07-14T19:00:02.487 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:00:02.488 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:00:02.488 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-14T19:00:02.488 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-14T19:00:02.488 INFO:teuthology.orchestra.run.smithi088.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0123478 s, 849 MB/s 2023-07-14T19:00:02.489 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 2023-07-14T19:00:02.489 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-14T19:00:02.891 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:00:02.891 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:00:02.892 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:02.892 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T19:00:02.892 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:00:02.892 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi078 2023-07-14T19:00:02.893 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:02.893 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:02.893 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:02.893 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:02.893 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:02.894 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi078 to 5722M 2023-07-14T19:00:02.894 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:02.894 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:02.894 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:02.895 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:02 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:02.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_2 2023-07-14T19:00:03.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:00:03.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:00:03.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:03.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T19:00:03.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:00:03.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi078 2023-07-14T19:00:03.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:03.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:03.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:03.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:03.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:03.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi078 to 5722M 2023-07-14T19:00:03.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:03.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:03.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:03.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:03.051 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:00:02.718+0000 7f3c7e9fe700 -1 osd.6 0 waiting for initial osdmap 2023-07-14T19:00:03.051 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:00:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:00:02.727+0000 7f3c7a7f4700 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:00:03.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-07-14T19:00:03.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: osdmap e43: 7 total, 6 up, 7 in 2023-07-14T19:00:03.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:03.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T19:00:03.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:00:03.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi078 2023-07-14T19:00:03.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:03.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:03.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:03.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:03.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:03.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi078 to 5722M 2023-07-14T19:00:03.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:03.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:03.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:03.135 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:02 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:03.518 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:00:04.022 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:03 smithi088 ceph-mon[128973]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T19:00:04.022 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:03 smithi088 ceph-mon[128973]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:00:04.022 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:04.022 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:03 smithi078 ceph-mon[127425]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T19:00:04.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:03 smithi078 ceph-mon[127425]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:00:04.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:04.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:04.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:03 smithi102 ceph-mon[129280]: from='osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T19:00:04.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:03 smithi102 ceph-mon[129280]: osdmap e44: 7 total, 6 up, 7 in 2023-07-14T19:00:04.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:04.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:04.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 -- 172.21.15.88:0/2675575190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 msgr2=0x7ff958107de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:04.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/2675575190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 0x7ff958107de0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b0a0 tx=0x7ff95000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:00:04.734 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 -- 172.21.15.88:0/2675575190 shutdown_connections 2023-07-14T19:00:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/2675575190 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 0x7ff958107de0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/2675575190 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9581007a0 0x7ff958102b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:04.735 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/2675575190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9581030c0 0x7ff9581054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 -- 172.21.15.88:0/2675575190 >> 172.21.15.88:0/2675575190 conn(0x7ff9580fa1e0 msgr2=0x7ff9580fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:00:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 -- 172.21.15.88:0/2675575190 shutdown_connections 2023-07-14T19:00:04.736 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 -- 172.21.15.88:0/2675575190 wait complete. 2023-07-14T19:00:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 Processor -- start 2023-07-14T19:00:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.728+0000 7ff95f9f7700 1 -- start start 2023-07-14T19:00:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95f9f7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9581007a0 0x7ff958195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:04.737 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95f9f7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9581030c0 0x7ff958195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95f9f7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 0x7ff95819cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95f9f7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95819d4d0 con 0x7ff9581007a0 2023-07-14T19:00:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95f9f7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95819d610 con 0x7ff9581059e0 2023-07-14T19:00:04.738 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95f9f7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95819d7a0 con 0x7ff9581030c0 2023-07-14T19:00:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 0x7ff95819cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:04.739 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 0x7ff95819cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60002/0 (socket says 172.21.15.88:60002) 2023-07-14T19:00:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 -- 172.21.15.88:0/3070307584 learned_addr learned my addr 172.21.15.88:0/3070307584 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T19:00:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95d793700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9581007a0 0x7ff958195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95cf92700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9581030c0 0x7ff958195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:04.740 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 -- 172.21.15.88:0/3070307584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9581030c0 msgr2=0x7ff958195e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9581030c0 0x7ff958195e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 -- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9581007a0 msgr2=0x7ff958195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9581007a0 0x7ff958195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:04.741 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 -- 172.21.15.88:0/3070307584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff950009d00 con 0x7ff9581059e0 2023-07-14T19:00:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95d793700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9581007a0 0x7ff958195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:00:04.742 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.729+0000 7ff95df94700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 0x7ff95819cf00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b780 tx=0x7ff95000bcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:00:04.743 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.732+0000 7ff94e7fc700 1 -- 172.21.15.88:0/3070307584 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95000e040 con 0x7ff9581059e0 2023-07-14T19:00:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.732+0000 7ff94e7fc700 1 -- 172.21.15.88:0/3070307584 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff950009480 con 0x7ff9581059e0 2023-07-14T19:00:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.732+0000 7ff94e7fc700 1 -- 172.21.15.88:0/3070307584 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9500052a0 con 0x7ff9581059e0 2023-07-14T19:00:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.732+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95819da20 con 0x7ff9581059e0 2023-07-14T19:00:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.732+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95819dde0 con 0x7ff9581059e0 2023-07-14T19:00:04.744 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.733+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9580fbd50 con 0x7ff9581059e0 2023-07-14T19:00:04.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.734+0000 7ff94e7fc700 1 -- 172.21.15.88:0/3070307584 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff95001d480 con 0x7ff9581059e0 2023-07-14T19:00:04.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.734+0000 7ff94e7fc700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff9440a11d0 0x7ff9440a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:04.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.735+0000 7ff94e7fc700 1 -- 172.21.15.88:0/3070307584 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 4965+0+0 (secure 0 0 0) 0x7ff9500ccf20 con 0x7ff9581059e0 2023-07-14T19:00:04.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.735+0000 7ff95d793700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff9440a11d0 0x7ff9440a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:04.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.735+0000 7ff95d793700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff9440a11d0 0x7ff9440a3680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff954011490 tx=0x7ff954009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:00:04.745 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.739+0000 7ff94e7fc700 1 -- 172.21.15.88:0/3070307584 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff950097410 con 0x7ff9581059e0 2023-07-14T19:00:04.938 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:04.934+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff958060a80 con 0x7ff9440a11d0 2023-07-14T19:00:05.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:04 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T19:00:05.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:04 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T19:00:05.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:04 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:05.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:04 smithi078 ceph-mon[127425]: osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] boot 2023-07-14T19:00:05.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:04 smithi078 ceph-mon[127425]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:00:05.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:04 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:05.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:04 smithi078 ceph-mon[127425]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:00:05.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:04 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T19:00:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:04 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T19:00:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:04 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:04 smithi102 ceph-mon[129280]: osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] boot 2023-07-14T19:00:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:04 smithi102 ceph-mon[129280]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:00:05.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:04 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:05.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:04 smithi102 ceph-mon[129280]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:00:05.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:04 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T19:00:05.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:04 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T19:00:05.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:04 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:05.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:04 smithi088 ceph-mon[128973]: osd.6 [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] boot 2023-07-14T19:00:05.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:04 smithi088 ceph-mon[128973]: osdmap e45: 7 total, 7 up, 7 in 2023-07-14T19:00:05.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:04 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-07-14T19:00:05.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:04 smithi088 ceph-mon[128973]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-07-14T19:00:06.000 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:05 smithi088 ceph-mon[128973]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:00:06.001 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:05 smithi088 ceph-mon[128973]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:06.001 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:06.001 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:06.001 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:06.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:05 smithi078 ceph-mon[127425]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:00:06.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:05 smithi078 ceph-mon[127425]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:06.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:06.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:06.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:06.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:05 smithi102 ceph-mon[129280]: osdmap e46: 7 total, 7 up, 7 in 2023-07-14T19:00:06.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:05 smithi102 ceph-mon[129280]: from='client.24299 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:06.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:06.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:06.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:06.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:06 smithi088 ceph-mon[128973]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:07.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:06 smithi078 ceph-mon[127425]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:07.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:06 smithi102 ceph-mon[129280]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:08.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:07 smithi078 ceph-mon[127425]: from='client.? 172.21.15.88:0/1487484284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]: dispatch 2023-07-14T19:00:08.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:07 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]: dispatch 2023-07-14T19:00:08.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:07 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]': finished 2023-07-14T19:00:08.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:07 smithi078 ceph-mon[127425]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:00:08.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:07 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:08.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:07 smithi102 ceph-mon[129280]: from='client.? 172.21.15.88:0/1487484284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]: dispatch 2023-07-14T19:00:08.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:07 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]: dispatch 2023-07-14T19:00:08.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:07 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]': finished 2023-07-14T19:00:08.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:07 smithi102 ceph-mon[129280]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:00:08.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:08.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:07 smithi088 ceph-mon[128973]: from='client.? 172.21.15.88:0/1487484284' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]: dispatch 2023-07-14T19:00:08.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:07 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]: dispatch 2023-07-14T19:00:08.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:07 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "603c8010-0bcc-4cb4-9e64-308839b69ea1"}]': finished 2023-07-14T19:00:08.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:07 smithi088 ceph-mon[128973]: osdmap e47: 8 total, 7 up, 8 in 2023-07-14T19:00:08.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:07 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:09.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:08 smithi078 ceph-mon[127425]: from='client.? 172.21.15.88:0/1915100168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:09.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:08 smithi078 ceph-mon[127425]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:09.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:08 smithi102 ceph-mon[129280]: from='client.? 172.21.15.88:0/1915100168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:09.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:08 smithi102 ceph-mon[129280]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:09.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:08 smithi088 ceph-mon[128973]: from='client.? 172.21.15.88:0/1915100168' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:09.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:08 smithi088 ceph-mon[128973]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:11.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:11 smithi088 ceph-mon[128973]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:11 smithi078 ceph-mon[127425]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:11.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:11 smithi102 ceph-mon[129280]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:13.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:13 smithi088 ceph-mon[128973]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:13.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:13 smithi078 ceph-mon[127425]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:13.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:13 smithi102 ceph-mon[129280]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:14.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:00:14.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:14.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:00:14.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:14.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-07-14T19:00:14.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:15.459 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:15 smithi088 ceph-mon[128973]: Deploying daemon osd.7 on smithi088 2023-07-14T19:00:15.460 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:15 smithi088 ceph-mon[128973]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:15.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:15 smithi078 ceph-mon[127425]: Deploying daemon osd.7 on smithi088 2023-07-14T19:00:15.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:15 smithi078 ceph-mon[127425]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:15.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:15 smithi102 ceph-mon[129280]: Deploying daemon osd.7 on smithi088 2023-07-14T19:00:15.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:15 smithi102 ceph-mon[129280]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:16.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:16 smithi102 ceph-mon[129280]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:16.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:16 smithi088 ceph-mon[128973]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:17.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:16 smithi078 ceph-mon[127425]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:19.585 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:19 smithi088 ceph-mon[128973]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:19.585 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:19.586 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:19.586 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:19.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:19 smithi078 ceph-mon[127425]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:19.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:19.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:19.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:19.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:19 smithi102 ceph-mon[129280]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:19.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:19.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:19.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:20.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:20 smithi088 ceph-mon[128973]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:20.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:20 smithi078 ceph-mon[127425]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:20.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:20 smithi102 ceph-mon[129280]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:21.304 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 7 on host 'smithi088' 2023-07-14T19:00:21.305 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.300+0000 7ff94e7fc700 1 -- 172.21.15.88:0/3070307584 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff958060a80 con 0x7ff9440a11d0 2023-07-14T19:00:21.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff9440a11d0 msgr2=0x7ff9440a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:21.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff9440a11d0 0x7ff9440a3680 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff954011490 tx=0x7ff954009260 comp rx=0 tx=0).stop 2023-07-14T19:00:21.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 msgr2=0x7ff95819cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:21.306 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 0x7ff95819cf00 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff95000b780 tx=0x7ff95000bcc0 comp rx=0 tx=0).stop 2023-07-14T19:00:21.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 shutdown_connections 2023-07-14T19:00:21.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff9440a11d0 0x7ff9440a3680 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:21.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff9581059e0 0x7ff95819cf00 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:21.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff9581007a0 0x7ff958195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:21.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 --2- 172.21.15.88:0/3070307584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9581030c0 0x7ff958195e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:21.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 >> 172.21.15.88:0/3070307584 conn(0x7ff9580fa1e0 msgr2=0x7ff958101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:00:21.307 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 shutdown_connections 2023-07-14T19:00:21.308 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:00:21.302+0000 7ff95f9f7700 1 -- 172.21.15.88:0/3070307584 wait complete. 2023-07-14T19:00:21.796 DEBUG:teuthology.orchestra.run.smithi088:osd.7> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.7.service 2023-07-14T19:00:21.798 INFO:tasks.cephadm:Deploying osd.8 on smithi102 with /dev/vg_nvme/lv_2... 2023-07-14T19:00:21.799 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-07-14T19:00:21.830 INFO:journalctl@ceph.osd.7.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T19:00:22.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:22.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:22.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:22.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:22.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:22.167 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.168 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.168 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.168 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:22.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:22.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:22.370 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:00:22.958 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:00:22 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:00:22.459+0000 7f7996288640 -1 osd.7 0 log_to_monitors true 2023-07-14T19:00:23.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:22 smithi088 ceph-mon[128973]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:23.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:22 smithi088 ceph-mon[128973]: from='osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:00:23.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:22 smithi088 ceph-mon[128973]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:00:23.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:22 smithi078 ceph-mon[127425]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:23.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:22 smithi078 ceph-mon[127425]: from='osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:00:23.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:22 smithi078 ceph-mon[127425]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:00:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:22 smithi102 ceph-mon[129280]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:22 smithi102 ceph-mon[129280]: from='osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:00:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:22 smithi102 ceph-mon[129280]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-07-14T19:00:23.951 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-07-14T19:00:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-07-14T19:00:23.952 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2023-07-14T19:00:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2023-07-14T19:00:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0110977 s, 945 MB/s 2023-07-14T19:00:23.952 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 2023-07-14T19:00:23.952 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2023-07-14T19:00:24.212 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:00:23 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:00:23.968+0000 7f798b900700 -1 osd.7 0 waiting for initial osdmap 2023-07-14T19:00:24.213 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:00:23 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:00:23.976+0000 7f79876f6700 -1 osd.7 49 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:00:24.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:23 smithi088 ceph-mon[128973]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:00:24.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:23 smithi088 ceph-mon[128973]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:00:24.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:24.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:23 smithi088 ceph-mon[128973]: from='osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:00:24.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:23 smithi088 ceph-mon[128973]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:00:24.251 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:23 smithi102 ceph-mon[129280]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:00:24.252 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:23 smithi102 ceph-mon[129280]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:00:24.252 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:24.252 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:23 smithi102 ceph-mon[129280]: from='osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:00:24.252 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:23 smithi102 ceph-mon[129280]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:00:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:23 smithi078 ceph-mon[127425]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-07-14T19:00:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:23 smithi078 ceph-mon[127425]: osdmap e48: 8 total, 7 up, 8 in 2023-07-14T19:00:24.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:24.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:23 smithi078 ceph-mon[127425]: from='osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:00:24.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:23 smithi078 ceph-mon[127425]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:00:24.324 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi102:vg_nvme/lv_2 2023-07-14T19:00:24.877 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:00:25.261 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T19:00:25.261 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:00:25.261 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:25.262 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:25.262 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:25.262 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi088 2023-07-14T19:00:25.262 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.262 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.262 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-14T19:00:25.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.263 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:25.264 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:25.264 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T19:00:25.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:00:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi088 2023-07-14T19:00:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-14T19:00:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:25.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T19:00:25.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: osdmap e49: 8 total, 7 up, 8 in 2023-07-14T19:00:25.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:25.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:25.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-07-14T19:00:25.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi088 2023-07-14T19:00:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi088 to 5722M 2023-07-14T19:00:25.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:25.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:25.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:25.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:26.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 -- 172.21.15.102:0/1575325357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd140febb0 msgr2=0x7efd140fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:26.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 --2- 172.21.15.102:0/1575325357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd140febb0 0x7efd140fefc0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efd08009990 tx=0x7efd08009ca0 comp rx=0 tx=0).stop 2023-07-14T19:00:26.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 -- 172.21.15.102:0/1575325357 shutdown_connections 2023-07-14T19:00:26.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 --2- 172.21.15.102:0/1575325357 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd14100740 0x7efd14107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:26.048 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 --2- 172.21.15.102:0/1575325357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd140febb0 0x7efd140fefc0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:26.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 --2- 172.21.15.102:0/1575325357 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd140ffdb0 0x7efd14100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:26.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 -- 172.21.15.102:0/1575325357 >> 172.21.15.102:0/1575325357 conn(0x7efd140fa1e0 msgr2=0x7efd140fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:00:26.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 -- 172.21.15.102:0/1575325357 shutdown_connections 2023-07-14T19:00:26.049 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.043+0000 7efd18e7d700 1 -- 172.21.15.102:0/1575325357 wait complete. 2023-07-14T19:00:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd18e7d700 1 Processor -- start 2023-07-14T19:00:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd18e7d700 1 -- start start 2023-07-14T19:00:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd18e7d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd140febb0 0x7efd141958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd18e7d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd140ffdb0 0x7efd14195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd18e7d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd14100740 0x7efd1419ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd18e7d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1419d440 con 0x7efd140febb0 2023-07-14T19:00:26.051 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd18e7d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1419d580 con 0x7efd140ffdb0 2023-07-14T19:00:26.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd18e7d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1419d710 con 0x7efd14100740 2023-07-14T19:00:26.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd14100740 0x7efd1419ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:26.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd14100740 0x7efd1419ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52264/0 (socket says 172.21.15.102:52264) 2023-07-14T19:00:26.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 -- 172.21.15.102:0/391669708 learned_addr learned my addr 172.21.15.102:0/391669708 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T19:00:26.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd11d9b700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd140ffdb0 0x7efd14195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:26.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd1259c700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd140febb0 0x7efd141958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:26.052 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 -- 172.21.15.102:0/391669708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd140ffdb0 msgr2=0x7efd14195df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd140ffdb0 0x7efd14195df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 -- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd140febb0 msgr2=0x7efd141958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd140febb0 0x7efd141958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 -- 172.21.15.102:0/391669708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd080095f0 con 0x7efd14100740 2023-07-14T19:00:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd1259c700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd140febb0 0x7efd141958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:00:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.044+0000 7efd12d9d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd14100740 0x7efd1419ce70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efd0c007f10 tx=0x7efd0c00d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:00:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.045+0000 7efd037fe700 1 -- 172.21.15.102:0/391669708 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c017070 con 0x7efd14100740 2023-07-14T19:00:26.053 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.045+0000 7efd037fe700 1 -- 172.21.15.102:0/391669708 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efd0c00f040 con 0x7efd14100740 2023-07-14T19:00:26.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.045+0000 7efd037fe700 1 -- 172.21.15.102:0/391669708 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd0c005710 con 0x7efd14100740 2023-07-14T19:00:26.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.045+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1419d9f0 con 0x7efd14100740 2023-07-14T19:00:26.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.045+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1419dea0 con 0x7efd14100740 2023-07-14T19:00:26.054 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.046+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1410a220 con 0x7efd14100740 2023-07-14T19:00:26.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.047+0000 7efd037fe700 1 -- 172.21.15.102:0/391669708 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efd0c004ad0 con 0x7efd14100740 2023-07-14T19:00:26.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.047+0000 7efd037fe700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efcfc0a11d0 0x7efcfc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:26.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.048+0000 7efd037fe700 1 -- 172.21.15.102:0/391669708 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 5426+0+0 (secure 0 0 0) 0x7efd0c0cb6b0 con 0x7efd14100740 2023-07-14T19:00:26.056 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.048+0000 7efd1259c700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efcfc0a11d0 0x7efcfc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:26.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.048+0000 7efd1259c700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efcfc0a11d0 0x7efcfc0a3680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd0800b5c0 tx=0x7efd08005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:00:26.057 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.052+0000 7efd037fe700 1 -- 172.21.15.102:0/391669708 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efd0c0958a0 con 0x7efd14100740 2023-07-14T19:00:26.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:25 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T19:00:26.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:25 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T19:00:26.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:26.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:25 smithi102 ceph-mon[129280]: osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] boot 2023-07-14T19:00:26.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:25 smithi102 ceph-mon[129280]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:00:26.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:26.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:25 smithi102 ceph-mon[129280]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:00:26.250 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:26.245+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7efd140688d0 con 0x7efcfc0a11d0 2023-07-14T19:00:26.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:25 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T19:00:26.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:25 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T19:00:26.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:26.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:25 smithi078 ceph-mon[127425]: osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] boot 2023-07-14T19:00:26.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:25 smithi078 ceph-mon[127425]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:00:26.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:26.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:25 smithi078 ceph-mon[127425]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:00:26.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:25 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T19:00:26.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:25 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T19:00:26.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:26.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:25 smithi088 ceph-mon[128973]: osd.7 [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] boot 2023-07-14T19:00:26.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:25 smithi088 ceph-mon[128973]: osdmap e50: 8 total, 8 up, 8 in 2023-07-14T19:00:26.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-07-14T19:00:26.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:25 smithi088 ceph-mon[128973]: osdmap e51: 8 total, 8 up, 8 in 2023-07-14T19:00:27.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:26 smithi078 ceph-mon[127425]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:27.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:26 smithi078 ceph-mon[127425]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:27.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:27.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:27.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:27.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:26 smithi078 ceph-mon[127425]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:00:27.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:26 smithi102 ceph-mon[129280]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:27.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:26 smithi102 ceph-mon[129280]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:27.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:27.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:27.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:27.350 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:26 smithi102 ceph-mon[129280]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:00:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:26 smithi088 ceph-mon[128973]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:26 smithi088 ceph-mon[128973]: from='client.24286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:27.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:26 smithi088 ceph-mon[128973]: osdmap e52: 8 total, 8 up, 8 in 2023-07-14T19:00:28.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:28 smithi102 ceph-mon[129280]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:00:28.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:28 smithi102 ceph-mon[129280]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:28 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/2400762763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]: dispatch 2023-07-14T19:00:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:28 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]: dispatch 2023-07-14T19:00:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:28 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]': finished 2023-07-14T19:00:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:28 smithi102 ceph-mon[129280]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:00:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:28.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:28 smithi088 ceph-mon[128973]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:00:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:28 smithi088 ceph-mon[128973]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:28 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/2400762763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]: dispatch 2023-07-14T19:00:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:28 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]: dispatch 2023-07-14T19:00:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:28 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]': finished 2023-07-14T19:00:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:28 smithi088 ceph-mon[128973]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:00:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:29.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:28 smithi078 ceph-mon[127425]: osdmap e53: 8 total, 8 up, 8 in 2023-07-14T19:00:29.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:28 smithi078 ceph-mon[127425]: pgmap v162: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:29.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:28 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/2400762763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]: dispatch 2023-07-14T19:00:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:28 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]: dispatch 2023-07-14T19:00:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:28 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "560f9ecc-e1d9-472d-af8b-ac1575902103"}]': finished 2023-07-14T19:00:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:28 smithi078 ceph-mon[127425]: osdmap e54: 9 total, 8 up, 9 in 2023-07-14T19:00:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:29.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:29 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/4203752758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:29.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:29 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/4203752758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:30.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:29 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/4203752758' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:30.548 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:00:30 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:00:30.067+0000 7ff2ec4b3700 -1 calc_pg_upmaps abort due to max <= 0 2023-07-14T19:00:30.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:30 smithi102 ceph-mon[129280]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:30.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:30 smithi088 ceph-mon[128973]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:31.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:30 smithi078 ceph-mon[127425]: pgmap v164: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:33.412 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:33 smithi102 ceph-mon[129280]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:33.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:33 smithi088 ceph-mon[128973]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:33.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:33 smithi078 ceph-mon[127425]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:35.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:35 smithi102 ceph-mon[129280]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:35.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:00:35.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:35.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:35 smithi088 ceph-mon[128973]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:35.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:00:35.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:35.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:35 smithi078 ceph-mon[127425]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:35.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2023-07-14T19:00:35.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:36.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:36 smithi102 ceph-mon[129280]: Deploying daemon osd.8 on smithi102 2023-07-14T19:00:36.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:36 smithi088 ceph-mon[128973]: Deploying daemon osd.8 on smithi102 2023-07-14T19:00:36.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:36 smithi078 ceph-mon[127425]: Deploying daemon osd.8 on smithi102 2023-07-14T19:00:37.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:37 smithi088 ceph-mon[128973]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:37.494 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:37 smithi102 ceph-mon[129280]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:37.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:37 smithi078 ceph-mon[127425]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:39.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:39 smithi102 ceph-mon[129280]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:39.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:39 smithi088 ceph-mon[128973]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:39.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:39 smithi078 ceph-mon[127425]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:40.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:40 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:40.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:40 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:40.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:40 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:40.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:40 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:40.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:40 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:40.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:40 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:40.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:40 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:40.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:40 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:40.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:40 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:41.578 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:41 smithi102 ceph-mon[129280]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:41.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:41 smithi088 ceph-mon[128973]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:41.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:41 smithi078 ceph-mon[127425]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:41.905 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.900+0000 7efd037fe700 1 -- 172.21.15.102:0/391669708 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7efd140688d0 con 0x7efcfc0a11d0 2023-07-14T19:00:41.906 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 8 on host 'smithi102' 2023-07-14T19:00:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efcfc0a11d0 msgr2=0x7efcfc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efcfc0a11d0 0x7efcfc0a3680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efd0800b5c0 tx=0x7efd08005fb0 comp rx=0 tx=0).stop 2023-07-14T19:00:41.908 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd14100740 msgr2=0x7efd1419ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd14100740 0x7efd1419ce70 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7efd0c007f10 tx=0x7efd0c00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:00:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 shutdown_connections 2023-07-14T19:00:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efcfc0a11d0 0x7efcfc0a3680 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:41.909 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd14100740 0x7efd1419ce70 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:41.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd140febb0 0x7efd141958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:41.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 --2- 172.21.15.102:0/391669708 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd140ffdb0 0x7efd14195df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:41.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 >> 172.21.15.102:0/391669708 conn(0x7efd140fa1e0 msgr2=0x7efd140681c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:00:41.910 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 shutdown_connections 2023-07-14T19:00:41.911 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:00:41.903+0000 7efd18e7d700 1 -- 172.21.15.102:0/391669708 wait complete. 2023-07-14T19:00:42.365 DEBUG:teuthology.orchestra.run.smithi102:osd.8> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.8.service 2023-07-14T19:00:42.367 INFO:tasks.cephadm:Deploying osd.9 on smithi078 with /dev/vg_nvme/lv_1... 2023-07-14T19:00:42.367 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:00:42.398 INFO:journalctl@ceph.osd.8.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T19:00:42.939 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:00:42.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:42.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:42.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:42.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:42.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:42.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:42.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:42.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:42.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:42 smithi088 ceph-mon[128973]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:43.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:43.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:43.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:42 smithi078 ceph-mon[127425]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:43.050 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.051 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.051 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:43.051 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:43.051 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.051 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.051 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:43.052 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:00:43.052 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:42 smithi102 ceph-mon[129280]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:43.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:00:43 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:00:43.259+0000 7fe6f013a640 -1 osd.8 0 log_to_monitors true 2023-07-14T19:00:44.208 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:43 smithi102 ceph-mon[129280]: from='osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:00:44.209 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:43 smithi102 ceph-mon[129280]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:00:44.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:43 smithi088 ceph-mon[128973]: from='osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:00:44.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:43 smithi088 ceph-mon[128973]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:00:44.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:43 smithi078 ceph-mon[127425]: from='osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:00:44.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:43 smithi078 ceph-mon[127425]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2023-07-14T19:00:44.411 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:00:44.412 INFO:teuthology.orchestra.run.smithi078.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:00:44.412 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10+0 records in 2023-07-14T19:00:44.412 INFO:teuthology.orchestra.run.smithi078.stderr:10+0 records out 2023-07-14T19:00:44.413 INFO:teuthology.orchestra.run.smithi078.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128217 s, 818 MB/s 2023-07-14T19:00:44.413 INFO:teuthology.orchestra.run.smithi078.stderr:--> Zapping successful for: 2023-07-14T19:00:44.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi078:vg_nvme/lv_1 2023-07-14T19:00:45.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:00:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:00:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:00:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:00:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:45.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:00:45.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:00:45.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:45.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:00:45.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:00:45.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:45.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:45.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:45.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.340 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:00:45.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:00:44.904+0000 7fe6e57b2700 -1 osd.8 0 waiting for initial osdmap 2023-07-14T19:00:45.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:00:44.913+0000 7fe6dfda5700 -1 osd.8 56 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:00:45.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2023-07-14T19:00:45.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: osdmap e55: 9 total, 8 up, 9 in 2023-07-14T19:00:45.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:45.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:00:45.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:00:45.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-07-14T19:00:45.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:00:45.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:45.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:45.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:00:45.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:00:46.126 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi102 2023-07-14T19:00:46.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi102 to 5722M 2023-07-14T19:00:46.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T19:00:46.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:00:46.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] boot 2023-07-14T19:00:46.127 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:00:46.128 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:45 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi102 2023-07-14T19:00:46.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi102 to 5722M 2023-07-14T19:00:46.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T19:00:46.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:00:46.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] boot 2023-07-14T19:00:46.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:00:46.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:45 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi102 2023-07-14T19:00:46.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi102 to 5722M 2023-07-14T19:00:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T19:00:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: osdmap e56: 9 total, 8 up, 9 in 2023-07-14T19:00:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: osd.8 [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] boot 2023-07-14T19:00:46.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: osdmap e57: 9 total, 9 up, 9 in 2023-07-14T19:00:46.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:45 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2023-07-14T19:00:46.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/2227851199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080febb0 msgr2=0x7f0f080fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:46.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/2227851199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080febb0 0x7f0f080fefc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009990 tx=0x7f0ef0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:00:46.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/2227851199 shutdown_connections 2023-07-14T19:00:46.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/2227851199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f08100740 0x7f0f08107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:46.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/2227851199 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080febb0 0x7f0f080fefc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:46.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/2227851199 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080ffdb0 0x7f0f08100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/2227851199 >> 172.21.15.78:0/2227851199 conn(0x7f0f080fa1e0 msgr2=0x7f0f080fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/2227851199 shutdown_connections 2023-07-14T19:00:46.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.662+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/2227851199 wait complete. 2023-07-14T19:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f0d8c7700 1 Processor -- start 2023-07-14T19:00:46.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f0d8c7700 1 -- start start 2023-07-14T19:00:46.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f0d8c7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080febb0 0x7f0f08194860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:46.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f0d8c7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080ffdb0 0x7f0f08194da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:46.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f0d8c7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f08100740 0x7f0f0819be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:46.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f0d8c7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0819c3f0 con 0x7f0f080ffdb0 2023-07-14T19:00:46.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f0d8c7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0819c530 con 0x7f0f080febb0 2023-07-14T19:00:46.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f0d8c7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0819c6c0 con 0x7f0f08100740 2023-07-14T19:00:46.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f067fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080ffdb0 0x7f0f08194da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:46.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f067fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080ffdb0 0x7f0f08194da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:33910/0 (socket says 172.21.15.78:33910) 2023-07-14T19:00:46.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f067fc700 1 -- 172.21.15.78:0/3996239920 learned_addr learned my addr 172.21.15.78:0/3996239920 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:00:46.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.663+0000 7f0f077fe700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f08100740 0x7f0f0819be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f06ffd700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080febb0 0x7f0f08194860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f067fc700 1 -- 172.21.15.78:0/3996239920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f08100740 msgr2=0x7f0f0819be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:46.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f067fc700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f08100740 0x7f0f0819be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f067fc700 1 -- 172.21.15.78:0/3996239920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080febb0 msgr2=0x7f0f08194860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:00:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f067fc700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080febb0 0x7f0f08194860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:00:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f067fc700 1 -- 172.21.15.78:0/3996239920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ef00095f0 con 0x7f0f080ffdb0 2023-07-14T19:00:46.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f06ffd700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080febb0 0x7f0f08194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:00:46.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f067fc700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080ffdb0 0x7f0f08194da0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800b560 tx=0x7f0ef800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:00:46.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f0c8c5700 1 -- 172.21.15.78:0/3996239920 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8011830 con 0x7f0f080ffdb0 2023-07-14T19:00:46.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f0c8c5700 1 -- 172.21.15.78:0/3996239920 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0ef8011e20 con 0x7f0f080ffdb0 2023-07-14T19:00:46.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f0819c9a0 con 0x7f0f080ffdb0 2023-07-14T19:00:46.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f0c8c5700 1 -- 172.21.15.78:0/3996239920 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef8010dd0 con 0x7f0f080ffdb0 2023-07-14T19:00:46.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.664+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f0819cf70 con 0x7f0f080ffdb0 2023-07-14T19:00:46.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.665+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f08106250 con 0x7f0f080ffdb0 2023-07-14T19:00:46.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.666+0000 7f0f0c8c5700 1 -- 172.21.15.78:0/3996239920 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0ef8011990 con 0x7f0f080ffdb0 2023-07-14T19:00:46.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.667+0000 7f0f0c8c5700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0ef40a1250 0x7f0ef40a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:00:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.667+0000 7f0f06ffd700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0ef40a1250 0x7f0ef40a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:00:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.667+0000 7f0f0c8c5700 1 -- 172.21.15.78:0/3996239920 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f0ef80cba50 con 0x7f0f080ffdb0 2023-07-14T19:00:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.667+0000 7f0f06ffd700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0ef40a1250 0x7f0ef40a3700 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00095c0 tx=0x7f0ef0009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:00:46.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.671+0000 7f0f0c8c5700 1 -- 172.21.15.78:0/3996239920 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0ef8095a90 con 0x7f0f080ffdb0 2023-07-14T19:00:46.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:00:46.864+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f08060a80 con 0x7f0ef40a1250 2023-07-14T19:00:47.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:46 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T19:00:47.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:46 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T19:00:47.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:46 smithi088 ceph-mon[128973]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:47.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:46 smithi088 ceph-mon[128973]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:00:47.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:47.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:47.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:46 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T19:00:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:46 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T19:00:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:46 smithi078 ceph-mon[127425]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:46 smithi078 ceph-mon[127425]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:00:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:47.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:47.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:46 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T19:00:47.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:46 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T19:00:47.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:46 smithi102 ceph-mon[129280]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:47.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:46 smithi102 ceph-mon[129280]: osdmap e58: 9 total, 9 up, 9 in 2023-07-14T19:00:47.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:00:47.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:00:47.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:47.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:47 smithi078 ceph-mon[127425]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:47.961 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:47 smithi078 ceph-mon[127425]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:00:48.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:47 smithi088 ceph-mon[128973]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:48.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:47 smithi088 ceph-mon[128973]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:00:48.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:47 smithi102 ceph-mon[129280]: from='client.14421 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi078:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:00:48.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:47 smithi102 ceph-mon[129280]: osdmap e59: 9 total, 9 up, 9 in 2023-07-14T19:00:49.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:48 smithi088 ceph-mon[128973]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:49.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:48 smithi078 ceph-mon[127425]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:49.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:48 smithi102 ceph-mon[129280]: pgmap v178: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:50.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:49 smithi088 ceph-mon[128973]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:00:50.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:49 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2615888922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afa084a-aebe-4fbf-af3b-36fcbf682dc4"}]: dispatch 2023-07-14T19:00:50.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:49 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2615888922' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8afa084a-aebe-4fbf-af3b-36fcbf682dc4"}]': finished 2023-07-14T19:00:50.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:49 smithi088 ceph-mon[128973]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:00:50.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:49 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:00:50.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:49 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/908145444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:50.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:49 smithi078 ceph-mon[127425]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:00:50.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:49 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2615888922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afa084a-aebe-4fbf-af3b-36fcbf682dc4"}]: dispatch 2023-07-14T19:00:50.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:49 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2615888922' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8afa084a-aebe-4fbf-af3b-36fcbf682dc4"}]': finished 2023-07-14T19:00:50.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:49 smithi078 ceph-mon[127425]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:00:50.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:49 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:00:50.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:49 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/908145444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:50.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:49 smithi102 ceph-mon[129280]: osdmap e60: 9 total, 9 up, 9 in 2023-07-14T19:00:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:49 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2615888922' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8afa084a-aebe-4fbf-af3b-36fcbf682dc4"}]: dispatch 2023-07-14T19:00:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:49 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2615888922' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8afa084a-aebe-4fbf-af3b-36fcbf682dc4"}]': finished 2023-07-14T19:00:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:49 smithi102 ceph-mon[129280]: osdmap e61: 10 total, 9 up, 10 in 2023-07-14T19:00:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:49 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:00:50.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:49 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/908145444' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:00:51.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:50 smithi088 ceph-mon[128973]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:50 smithi078 ceph-mon[127425]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:51.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:50 smithi102 ceph-mon[129280]: pgmap v181: 1 pgs: 1 remapped+peering; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:00:53.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:53 smithi078 ceph-mon[127425]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-14T19:00:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:53 smithi088 ceph-mon[128973]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-14T19:00:53.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:53 smithi102 ceph-mon[129280]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 105 KiB/s, 0 objects/s recovering 2023-07-14T19:00:55.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:55 smithi078 ceph-mon[127425]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T19:00:55.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:55 smithi088 ceph-mon[128973]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T19:00:55.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:55 smithi102 ceph-mon[129280]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 89 KiB/s, 0 objects/s recovering 2023-07-14T19:00:56.684 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:00:56.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:56 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:56.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:56 smithi078 ceph-mon[127425]: Deploying daemon osd.9 on smithi078 2023-07-14T19:00:56.685 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:56 smithi078 ceph-mon[127425]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:00:56.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:00:56.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:56 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:56.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:56 smithi102 ceph-mon[129280]: Deploying daemon osd.9 on smithi078 2023-07-14T19:00:56.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:56 smithi102 ceph-mon[129280]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:00:56.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2023-07-14T19:00:56.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:56 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:00:56.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:56 smithi088 ceph-mon[128973]: Deploying daemon osd.9 on smithi078 2023-07-14T19:00:56.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:56 smithi088 ceph-mon[128973]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-07-14T19:00:59.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:00:59 smithi088 ceph-mon[128973]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:00:59.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:00:59 smithi078 ceph-mon[127425]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:00:59.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:00:59 smithi102 ceph-mon[129280]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2023-07-14T19:01:01.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:01 smithi078 ceph-mon[127425]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-07-14T19:01:01.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:01.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:01.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:01.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:01 smithi088 ceph-mon[128973]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-07-14T19:01:01.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:01.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:01.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:01.570 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:01 smithi102 ceph-mon[129280]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 53 KiB/s, 0 objects/s recovering 2023-07-14T19:01:01.570 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:01.570 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:01.571 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:03.167 INFO:teuthology.orchestra.run.smithi078.stdout:Created osd(s) 9 on host 'smithi078' 2023-07-14T19:01:03.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.163+0000 7f0f0c8c5700 1 -- 172.21.15.78:0/3996239920 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0f08060a80 con 0x7f0ef40a1250 2023-07-14T19:01:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0ef40a1250 msgr2=0x7f0ef40a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0ef40a1250 0x7f0ef40a3700 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0ef00095c0 tx=0x7f0ef0009510 comp rx=0 tx=0).stop 2023-07-14T19:01:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080ffdb0 msgr2=0x7f0f08194da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:03.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080ffdb0 0x7f0f08194da0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f0ef800b560 tx=0x7f0ef800b870 comp rx=0 tx=0).stop 2023-07-14T19:01:03.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 shutdown_connections 2023-07-14T19:01:03.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0ef40a1250 0x7f0ef40a3700 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:03.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0f080febb0 0x7f0f08194860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:03.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0f08100740 0x7f0f0819be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:03.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 --2- 172.21.15.78:0/3996239920 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0f080ffdb0 0x7f0f08194da0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:03.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 >> 172.21.15.78:0/3996239920 conn(0x7f0f080fa1e0 msgr2=0x7f0f08103390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:03.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 shutdown_connections 2023-07-14T19:01:03.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:03.165+0000 7f0f0d8c7700 1 -- 172.21.15.78:0/3996239920 wait complete. 2023-07-14T19:01:03.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:03.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:03.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:03.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.527 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.528 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:03.529 DEBUG:teuthology.orchestra.run.smithi078:osd.9> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.9.service 2023-07-14T19:01:03.531 INFO:tasks.cephadm:Deploying osd.10 on smithi088 with /dev/vg_nvme/lv_1... 2023-07-14T19:01:03.532 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:01:03.564 INFO:journalctl@ceph.osd.9.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T19:01:03.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:03.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:03.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:03.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:03.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:04.096 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:01:04.403 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:04 smithi078 ceph-mon[127425]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:01:04.404 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:01:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:01:04.131+0000 7f28aa902640 -1 osd.9 0 log_to_monitors true 2023-07-14T19:01:04.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:04 smithi088 ceph-mon[128973]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:01:04.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:04 smithi102 ceph-mon[129280]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2023-07-14T19:01:05.221 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:01:05 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:01:05.202+0000 7f289ff7a700 -1 osd.9 0 waiting for initial osdmap 2023-07-14T19:01:05.221 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:01:05 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:01:05.216+0000 7f2899d6c700 -1 osd.9 63 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:01:05.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:05 smithi078 ceph-mon[127425]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:01:05.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:05 smithi078 ceph-mon[127425]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:01:05.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:05 smithi078 ceph-mon[127425]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:01:05.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:05.222 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:05 smithi078 ceph-mon[127425]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T19:01:05.322 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:05 smithi088 ceph-mon[128973]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:01:05.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:05 smithi088 ceph-mon[128973]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:01:05.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:05 smithi088 ceph-mon[128973]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:01:05.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:05.323 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:05 smithi088 ceph-mon[128973]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T19:01:05.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:05 smithi102 ceph-mon[129280]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:01:05.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:05 smithi102 ceph-mon[129280]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2023-07-14T19:01:05.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:05 smithi102 ceph-mon[129280]: osdmap e62: 10 total, 9 up, 10 in 2023-07-14T19:01:05.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:05.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:05 smithi102 ceph-mon[129280]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]: dispatch 2023-07-14T19:01:05.791 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:01:05.791 INFO:teuthology.orchestra.run.smithi088.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:01:05.792 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10+0 records in 2023-07-14T19:01:05.792 INFO:teuthology.orchestra.run.smithi088.stderr:10+0 records out 2023-07-14T19:01:05.792 INFO:teuthology.orchestra.run.smithi088.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109007 s, 962 MB/s 2023-07-14T19:01:05.792 INFO:teuthology.orchestra.run.smithi088.stderr:--> Zapping successful for: 2023-07-14T19:01:06.250 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi088:vg_nvme/lv_1 2023-07-14T19:01:06.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T19:01:06.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:01:06.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:06.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:06.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.551 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:06.552 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:06.553 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T19:01:06.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:01:06.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:06.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:06.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:06.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:06.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi078", "root=default"]}]': finished 2023-07-14T19:01:06.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: osdmap e63: 10 total, 9 up, 10 in 2023-07-14T19:01:06.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:06.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:06.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:06.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:06.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:06.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:06.802 INFO:teuthology.orchestra.run.smithi088.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.b/config 2023-07-14T19:01:07.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T19:01:07.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T19:01:07.307 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi078 2023-07-14T19:01:07.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi078 to 4292M 2023-07-14T19:01:07.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:01:07.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:07.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] boot 2023-07-14T19:01:07.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:01:07.308 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:07 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:07.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T19:01:07.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T19:01:07.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi078 2023-07-14T19:01:07.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi078 to 4292M 2023-07-14T19:01:07.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:01:07.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:07.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] boot 2023-07-14T19:01:07.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:01:07.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:07 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:07.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T19:01:07.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T19:01:07.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi078 2023-07-14T19:01:07.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi078 to 4292M 2023-07-14T19:01:07.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 245 MiB used, 804 GiB / 805 GiB avail 2023-07-14T19:01:07.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:07.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: osd.9 [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] boot 2023-07-14T19:01:07.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: osdmap e64: 10 total, 10 up, 10 in 2023-07-14T19:01:07.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2023-07-14T19:01:08.040 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 -- 172.21.15.88:0/3562347890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 msgr2=0x7f637c0fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 --2- 172.21.15.88:0/3562347890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 0x7f637c0fefc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f636c009990 tx=0x7f636c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:01:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 -- 172.21.15.88:0/3562347890 shutdown_connections 2023-07-14T19:01:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 --2- 172.21.15.88:0/3562347890 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 0x7f637c0fefc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 --2- 172.21.15.88:0/3562347890 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f637c100740 0x7f637c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 --2- 172.21.15.88:0/3562347890 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f637c0ffdb0 0x7f637c100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:08.041 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 -- 172.21.15.88:0/3562347890 >> 172.21.15.88:0/3562347890 conn(0x7f637c0fa1e0 msgr2=0x7f637c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 -- 172.21.15.88:0/3562347890 shutdown_connections 2023-07-14T19:01:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.035+0000 7f6383566700 1 -- 172.21.15.88:0/3562347890 wait complete. 2023-07-14T19:01:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6383566700 1 Processor -- start 2023-07-14T19:01:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6383566700 1 -- start start 2023-07-14T19:01:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6383566700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 0x7f637c069510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6383566700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f637c0ffdb0 0x7f637c19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:08.042 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6383566700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f637c100740 0x7f637c19cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6383566700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c19d420 con 0x7f637c0ffdb0 2023-07-14T19:01:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6383566700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c19d590 con 0x7f637c0febb0 2023-07-14T19:01:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6383566700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f637c19d750 con 0x7f637c100740 2023-07-14T19:01:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 0x7f637c069510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 0x7f637c069510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.88:60148/0 (socket says 172.21.15.88:60148) 2023-07-14T19:01:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 -- 172.21.15.88:0/702225912 learned_addr learned my addr 172.21.15.88:0/702225912 (peer_addr_for_me v2:172.21.15.88:0/0) 2023-07-14T19:01:08.043 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381b03700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f637c100740 0x7f637c19cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6380b01700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f637c0ffdb0 0x7f637c19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 -- 172.21.15.88:0/702225912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f637c100740 msgr2=0x7f637c19cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f637c100740 0x7f637c19cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 -- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f637c0ffdb0 msgr2=0x7f637c19a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f637c0ffdb0 0x7f637c19a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:08.044 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 -- 172.21.15.88:0/702225912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f636c0095f0 con 0x7f637c0febb0 2023-07-14T19:01:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6380b01700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f637c0ffdb0 0x7f637c19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:01:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.036+0000 7f6381302700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 0x7f637c069510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f636c0059d0 tx=0x7f636c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.037+0000 7f63727fc700 1 -- 172.21.15.88:0/702225912 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c01d070 con 0x7f637c0febb0 2023-07-14T19:01:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.037+0000 7f63727fc700 1 -- 172.21.15.88:0/702225912 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f636c00beb0 con 0x7f637c0febb0 2023-07-14T19:01:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.037+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c19d9d0 con 0x7f637c0febb0 2023-07-14T19:01:08.045 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.037+0000 7f63727fc700 1 -- 172.21.15.88:0/702225912 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f636c022e40 con 0x7f637c0febb0 2023-07-14T19:01:08.046 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.037+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c19de40 con 0x7f637c0febb0 2023-07-14T19:01:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.037+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c106280 con 0x7f637c0febb0 2023-07-14T19:01:08.047 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.038+0000 7f63727fc700 1 -- 172.21.15.88:0/702225912 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f636c022600 con 0x7f637c0febb0 2023-07-14T19:01:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.039+0000 7f63727fc700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f63680a11d0 0x7f63680a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.039+0000 7f63727fc700 1 -- 172.21.15.88:0/702225912 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 6261+0+0 (secure 0 0 0) 0x7f636c0cd7d0 con 0x7f637c0febb0 2023-07-14T19:01:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.039+0000 7f6380b01700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f63680a11d0 0x7f63680a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.040+0000 7f6380b01700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f63680a11d0 0x7f63680a3680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6378005fd0 tx=0x7f6378005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:08.048 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.043+0000 7f63727fc700 1 -- 172.21.15.88:0/702225912 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f636c0977b0 con 0x7f637c0febb0 2023-07-14T19:01:08.241 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:08.238+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f637c060a80 con 0x7f63680a11d0 2023-07-14T19:01:09.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:08 smithi088 ceph-mon[128973]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:01:09.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:08 smithi088 ceph-mon[128973]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:09.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:08 smithi088 ceph-mon[128973]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:09.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:01:09.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:01:09.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:09.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:08 smithi078 ceph-mon[127425]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:01:09.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:08 smithi078 ceph-mon[127425]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:09.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:08 smithi078 ceph-mon[127425]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:09.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:01:09.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:01:09.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:09.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:08 smithi102 ceph-mon[129280]: osdmap e65: 10 total, 10 up, 10 in 2023-07-14T19:01:09.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:08 smithi102 ceph-mon[129280]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:09.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:08 smithi102 ceph-mon[129280]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi088:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:09.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:01:09.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:01:09.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:11.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:11 smithi088 ceph-mon[128973]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:11.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:11 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]: dispatch 2023-07-14T19:01:11.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:11 smithi088 ceph-mon[128973]: from='client.? 172.21.15.88:0/2798116209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]: dispatch 2023-07-14T19:01:11.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:11 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]': finished 2023-07-14T19:01:11.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:11 smithi088 ceph-mon[128973]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:01:11.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:11 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:11.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:11 smithi078 ceph-mon[127425]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:11 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]: dispatch 2023-07-14T19:01:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:11 smithi078 ceph-mon[127425]: from='client.? 172.21.15.88:0/2798116209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]: dispatch 2023-07-14T19:01:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:11 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]': finished 2023-07-14T19:01:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:11 smithi078 ceph-mon[127425]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:01:11.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:11 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:11.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:11 smithi102 ceph-mon[129280]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:11.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:11 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]: dispatch 2023-07-14T19:01:11.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:11 smithi102 ceph-mon[129280]: from='client.? 172.21.15.88:0/2798116209' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]: dispatch 2023-07-14T19:01:11.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:11 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f150df7b-04f5-4103-bd4f-d93f48408f48"}]': finished 2023-07-14T19:01:11.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:11 smithi102 ceph-mon[129280]: osdmap e66: 11 total, 10 up, 11 in 2023-07-14T19:01:11.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:11 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:12.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:12 smithi088 ceph-mon[128973]: from='client.? 172.21.15.88:0/2482216008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:01:12.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:12 smithi078 ceph-mon[127425]: from='client.? 172.21.15.88:0/2482216008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:01:12.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:12 smithi102 ceph-mon[129280]: from='client.? 172.21.15.88:0/2482216008' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:01:13.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:13 smithi088 ceph-mon[128973]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:13.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:13 smithi078 ceph-mon[127425]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:13.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:13 smithi102 ceph-mon[129280]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:15.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:15 smithi088 ceph-mon[128973]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:15.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:15 smithi078 ceph-mon[127425]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:15.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:15 smithi102 ceph-mon[129280]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:16.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:16 smithi102 ceph-mon[129280]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:16.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:16 smithi088 ceph-mon[128973]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:17.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:16 smithi078 ceph-mon[127425]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:17.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:01:17.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:17.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:17 smithi088 ceph-mon[128973]: Deploying daemon osd.10 on smithi088 2023-07-14T19:01:17.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:01:17.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:17.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:17 smithi102 ceph-mon[129280]: Deploying daemon osd.10 on smithi088 2023-07-14T19:01:18.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2023-07-14T19:01:18.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:18.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:17 smithi078 ceph-mon[127425]: Deploying daemon osd.10 on smithi088 2023-07-14T19:01:18.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:18 smithi088 ceph-mon[128973]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:18.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:18 smithi102 ceph-mon[129280]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:19.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:18 smithi078 ceph-mon[127425]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:21.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:21 smithi088 ceph-mon[128973]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:21.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:21 smithi078 ceph-mon[127425]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:21.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:21 smithi102 ceph-mon[129280]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:23.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:23 smithi088 ceph-mon[128973]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:23.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:23.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:23.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:23.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:23 smithi102 ceph-mon[129280]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:23.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:23.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:23.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:23 smithi078 ceph-mon[127425]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:23.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:23.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:23.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:24.953 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.949+0000 7f63727fc700 1 -- 172.21.15.88:0/702225912 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f637c060a80 con 0x7f63680a11d0 2023-07-14T19:01:24.954 INFO:teuthology.orchestra.run.smithi088.stdout:Created osd(s) 10 on host 'smithi088' 2023-07-14T19:01:24.955 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f63680a11d0 msgr2=0x7f63680a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f63680a11d0 0x7f63680a3680 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6378005fd0 tx=0x7f6378005f60 comp rx=0 tx=0).stop 2023-07-14T19:01:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 msgr2=0x7f637c069510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 0x7f637c069510 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f636c0059d0 tx=0x7f636c004730 comp rx=0 tx=0).stop 2023-07-14T19:01:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 shutdown_connections 2023-07-14T19:01:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f63680a11d0 0x7f63680a3680 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f637c0febb0 0x7f637c069510 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:24.956 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f637c100740 0x7f637c19cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:24.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 --2- 172.21.15.88:0/702225912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f637c0ffdb0 0x7f637c19a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:24.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 >> 172.21.15.88:0/702225912 conn(0x7f637c0fa1e0 msgr2=0x7f637c1033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:24.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 shutdown_connections 2023-07-14T19:01:24.957 INFO:teuthology.orchestra.run.smithi088.stderr:2023-07-14T19:01:24.951+0000 7f6383566700 1 -- 172.21.15.88:0/702225912 wait complete. 2023-07-14T19:01:25.290 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:25 smithi088 ceph-mon[128973]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:25.290 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.291 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.292 DEBUG:teuthology.orchestra.run.smithi088:osd.10> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.10.service 2023-07-14T19:01:25.294 INFO:tasks.cephadm:Deploying osd.11 on smithi102 with /dev/vg_nvme/lv_1... 2023-07-14T19:01:25.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2023-07-14T19:01:25.324 INFO:journalctl@ceph.osd.10.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:25 smithi102 ceph-mon[129280]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:25 smithi078 ceph-mon[127425]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:25.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:25.858 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:01:26.418 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:01:26 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:01:26.104+0000 7f4887e5f640 -1 osd.10 0 log_to_monitors true 2023-07-14T19:01:26.692 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:26 smithi088 ceph-mon[128973]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:26.693 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:26 smithi088 ceph-mon[128973]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:01:26.693 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:26 smithi088 ceph-mon[128973]: from='osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:01:26.856 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:26 smithi102 ceph-mon[129280]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:26.856 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:26 smithi102 ceph-mon[129280]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:01:26.856 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:26 smithi102 ceph-mon[129280]: from='osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:01:27.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:26 smithi078 ceph-mon[127425]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:27.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:26 smithi078 ceph-mon[127425]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:01:27.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:26 smithi078 ceph-mon[127425]: from='osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2023-07-14T19:01:27.316 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_1 2023-07-14T19:01:27.317 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2023-07-14T19:01:27.317 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2023-07-14T19:01:27.317 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2023-07-14T19:01:27.317 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125377 s, 836 MB/s 2023-07-14T19:01:27.317 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2023-07-14T19:01:27.701 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch daemon add osd smithi102:vg_nvme/lv_1 2023-07-14T19:01:27.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:27 smithi102 ceph-mon[129280]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:01:27.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:27 smithi102 ceph-mon[129280]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:01:27.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:27 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:27.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:27 smithi102 ceph-mon[129280]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:01:27.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:27 smithi102 ceph-mon[129280]: from='osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:01:27.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:01:27 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:01:27.612+0000 7f487d4d7700 -1 osd.10 0 waiting for initial osdmap 2023-07-14T19:01:27.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:01:27 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:01:27.625+0000 7f48772c9700 -1 osd.10 68 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:01:27.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:27 smithi088 ceph-mon[128973]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:01:27.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:27 smithi088 ceph-mon[128973]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:01:27.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:27 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:27.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:27 smithi088 ceph-mon[128973]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:01:27.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:27 smithi088 ceph-mon[128973]: from='osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:01:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:27 smithi078 ceph-mon[127425]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2023-07-14T19:01:28.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:27 smithi078 ceph-mon[127425]: osdmap e67: 11 total, 10 up, 11 in 2023-07-14T19:01:28.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:27 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:28.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:27 smithi078 ceph-mon[127425]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:01:28.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:27 smithi078 ceph-mon[127425]: from='osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]: dispatch 2023-07-14T19:01:28.253 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:01:28.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T19:01:28.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:01:28.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:28.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:28.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi088 2023-07-14T19:01:28.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:28.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:28.734 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:28.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:28.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-14T19:01:28.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:28.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:28.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.735 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:28.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:28.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:28.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:28.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] boot 2023-07-14T19:01:28.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:01:28.736 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:28.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T19:01:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:01:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi088 2023-07-14T19:01:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:28.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:28.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:28.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:28.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-14T19:01:28.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:28.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:28.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:28.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:28.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:28.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:28.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:28.965 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] boot 2023-07-14T19:01:28.966 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:01:28.966 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:29.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi088", "root=default"]}]': finished 2023-07-14T19:01:29.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: osdmap e68: 11 total, 10 up, 11 in 2023-07-14T19:01:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi088 2023-07-14T19:01:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:29.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:29.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:29.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:29.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi088 to 4292M 2023-07-14T19:01:29.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:29.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:29.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:29.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:29.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:29.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:29.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:29.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:29.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 272 MiB used, 894 GiB / 894 GiB avail 2023-07-14T19:01:29.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: osd.10 [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] boot 2023-07-14T19:01:29.051 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: osdmap e69: 11 total, 11 up, 11 in 2023-07-14T19:01:29.052 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2023-07-14T19:01:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 -- 172.21.15.102:0/2552127916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238100ec0 msgr2=0x7f7238101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:29.455 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 --2- 172.21.15.102:0/2552127916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238100ec0 0x7f7238101310 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7234009990 tx=0x7f7234009ca0 comp rx=0 tx=0).stop 2023-07-14T19:01:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 -- 172.21.15.102:0/2552127916 shutdown_connections 2023-07-14T19:01:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 --2- 172.21.15.102:0/2552127916 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7238101850 0x7f7238108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 --2- 172.21.15.102:0/2552127916 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f72380ffcc0 0x7f72381000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 --2- 172.21.15.102:0/2552127916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238100ec0 0x7f7238101310 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 -- 172.21.15.102:0/2552127916 >> 172.21.15.102:0/2552127916 conn(0x7f72380fb2d0 msgr2=0x7f72380fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 -- 172.21.15.102:0/2552127916 shutdown_connections 2023-07-14T19:01:29.456 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.449+0000 7f723f6fb700 1 -- 172.21.15.102:0/2552127916 wait complete. 2023-07-14T19:01:29.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723f6fb700 1 Processor -- start 2023-07-14T19:01:29.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723f6fb700 1 -- start start 2023-07-14T19:01:29.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723f6fb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72380ffcc0 0x7f7238197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:29.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723f6fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7238100ec0 0x7f72381981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:29.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723f6fb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238101850 0x7f723819f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:29.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723f6fb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723819f800 con 0x7f7238100ec0 2023-07-14T19:01:29.457 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723f6fb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723819f940 con 0x7f72380ffcc0 2023-07-14T19:01:29.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723f6fb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723819fb00 con 0x7f7238101850 2023-07-14T19:01:29.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723dc98700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238101850 0x7f723819f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:29.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723dc98700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238101850 0x7f723819f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52406/0 (socket says 172.21.15.102:52406) 2023-07-14T19:01:29.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723dc98700 1 -- 172.21.15.102:0/4263405157 learned_addr learned my addr 172.21.15.102:0/4263405157 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T19:01:29.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.450+0000 7f723d497700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72380ffcc0 0x7f7238197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:29.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723cc96700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7238100ec0 0x7f72381981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:29.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723dc98700 1 -- 172.21.15.102:0/4263405157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72380ffcc0 msgr2=0x7f7238197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:29.458 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723dc98700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72380ffcc0 0x7f7238197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723dc98700 1 -- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7238100ec0 msgr2=0x7f72381981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723dc98700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7238100ec0 0x7f72381981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723dc98700 1 -- 172.21.15.102:0/4263405157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72340095f0 con 0x7f7238101850 2023-07-14T19:01:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723cc96700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7238100ec0 0x7f72381981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:01:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723dc98700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238101850 0x7f723819f230 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f723000b970 tx=0x7f723000bc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f722e7fc700 1 -- 172.21.15.102:0/4263405157 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f723000c930 con 0x7f7238101850 2023-07-14T19:01:29.459 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723819fde0 con 0x7f7238101850 2023-07-14T19:01:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f722e7fc700 1 -- 172.21.15.102:0/4263405157 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7230014440 con 0x7f7238101850 2023-07-14T19:01:29.460 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f722e7fc700 1 -- 172.21.15.102:0/4263405157 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72300075d0 con 0x7f7238101850 2023-07-14T19:01:29.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.451+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72381a02d0 con 0x7f7238101850 2023-07-14T19:01:29.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.452+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f721c005320 con 0x7f7238101850 2023-07-14T19:01:29.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.453+0000 7f722e7fc700 1 -- 172.21.15.102:0/4263405157 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f723000ca90 con 0x7f7238101850 2023-07-14T19:01:29.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.453+0000 7f722e7fc700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f72240a1150 0x7f72240a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:29.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.454+0000 7f722e7fc700 1 -- 172.21.15.102:0/4263405157 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6727+0+0 (secure 0 0 0) 0x7f7230006e40 con 0x7f7238101850 2023-07-14T19:01:29.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.454+0000 7f723d497700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f72240a1150 0x7f72240a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:29.461 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.454+0000 7f723d497700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f72240a1150 0x7f72240a3600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7238198b60 tx=0x7f7228009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:29.462 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.458+0000 7f722e7fc700 1 -- 172.21.15.102:0/4263405157 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f72300958d0 con 0x7f7238101850 2023-07-14T19:01:29.656 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:29.652+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f721c000bf0 con 0x7f72240a1150 2023-07-14T19:01:30.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:29 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T19:01:30.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:29 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T19:01:30.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:29 smithi088 ceph-mon[128973]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:01:30.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:01:30.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:01:30.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:30.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:29 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T19:01:30.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:29 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T19:01:30.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:29 smithi078 ceph-mon[127425]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:01:30.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:01:30.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:01:30.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:30.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:29 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T19:01:30.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:29 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T19:01:30.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:29 smithi102 ceph-mon[129280]: osdmap e70: 11 total, 11 up, 11 in 2023-07-14T19:01:30.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-07-14T19:01:30.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-07-14T19:01:30.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:31.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:30 smithi102 ceph-mon[129280]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:31.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:30 smithi102 ceph-mon[129280]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:31.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:30 smithi102 ceph-mon[129280]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:01:31.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:30 smithi088 ceph-mon[128973]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:31.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:30 smithi088 ceph-mon[128973]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:31.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:30 smithi088 ceph-mon[128973]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:01:31.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:30 smithi078 ceph-mon[127425]: from='client.24352 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:31.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:30 smithi078 ceph-mon[127425]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 299 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:31.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:30 smithi078 ceph-mon[127425]: osdmap e71: 11 total, 11 up, 11 in 2023-07-14T19:01:33.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:32 smithi088 ceph-mon[128973]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:01:33.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:32 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/2701893289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]: dispatch 2023-07-14T19:01:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:32 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]: dispatch 2023-07-14T19:01:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:32 smithi088 ceph-mon[128973]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]': finished 2023-07-14T19:01:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:32 smithi088 ceph-mon[128973]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:01:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:32 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:32 smithi088 ceph-mon[128973]: pgmap v214: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:01:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:32 smithi088 ceph-mon[128973]: from='client.? 172.21.15.102:0/3693057597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:01:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:32 smithi078 ceph-mon[127425]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:01:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:32 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/2701893289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]: dispatch 2023-07-14T19:01:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:32 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]: dispatch 2023-07-14T19:01:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:32 smithi078 ceph-mon[127425]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]': finished 2023-07-14T19:01:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:32 smithi078 ceph-mon[127425]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:01:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:32 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:32 smithi078 ceph-mon[127425]: pgmap v214: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:01:33.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:32 smithi078 ceph-mon[127425]: from='client.? 172.21.15.102:0/3693057597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:01:33.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:32 smithi102 ceph-mon[129280]: osdmap e72: 11 total, 11 up, 11 in 2023-07-14T19:01:33.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:32 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/2701893289' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]: dispatch 2023-07-14T19:01:33.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:32 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]: dispatch 2023-07-14T19:01:33.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:32 smithi102 ceph-mon[129280]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dc848771-b0b4-4e08-838a-25775af1cefb"}]': finished 2023-07-14T19:01:33.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:32 smithi102 ceph-mon[129280]: osdmap e73: 12 total, 11 up, 12 in 2023-07-14T19:01:33.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:32 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:33.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:32 smithi102 ceph-mon[129280]: pgmap v214: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:01:33.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:32 smithi102 ceph-mon[129280]: from='client.? 172.21.15.102:0/3693057597' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-07-14T19:01:34.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:33 smithi088 ceph-mon[128973]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-14T19:01:34.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:33 smithi078 ceph-mon[127425]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-14T19:01:34.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:33 smithi102 ceph-mon[129280]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2023-07-14T19:01:35.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:34 smithi088 ceph-mon[128973]: pgmap v215: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:01:35.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:34 smithi078 ceph-mon[127425]: pgmap v215: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:01:35.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:34 smithi102 ceph-mon[129280]: pgmap v215: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 2/6 objects degraded (33.333%) 2023-07-14T19:01:36.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:36 smithi102 ceph-mon[129280]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:01:36.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:36 smithi088 ceph-mon[128973]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:01:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:36 smithi078 ceph-mon[127425]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2023-07-14T19:01:37.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:37 smithi102 ceph-mon[129280]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-14T19:01:37.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:37 smithi102 ceph-mon[129280]: Cluster is now healthy 2023-07-14T19:01:37.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:37 smithi088 ceph-mon[128973]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-14T19:01:37.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:37 smithi088 ceph-mon[128973]: Cluster is now healthy 2023-07-14T19:01:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:37 smithi078 ceph-mon[127425]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2023-07-14T19:01:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:37 smithi078 ceph-mon[127425]: Cluster is now healthy 2023-07-14T19:01:38.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:38 smithi102 ceph-mon[129280]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T19:01:38.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:01:38.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:38 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:38.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:38 smithi088 ceph-mon[128973]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T19:01:38.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:01:38.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:38 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:39.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:38 smithi078 ceph-mon[127425]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2023-07-14T19:01:39.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2023-07-14T19:01:39.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:38 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:39.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:39 smithi102 ceph-mon[129280]: Deploying daemon osd.11 on smithi102 2023-07-14T19:01:39.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:39 smithi088 ceph-mon[128973]: Deploying daemon osd.11 on smithi102 2023-07-14T19:01:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:39 smithi078 ceph-mon[127425]: Deploying daemon osd.11 on smithi102 2023-07-14T19:01:40.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:40 smithi102 ceph-mon[129280]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T19:01:40.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:40 smithi088 ceph-mon[128973]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T19:01:41.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:40 smithi078 ceph-mon[127425]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-07-14T19:01:43.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:43 smithi088 ceph-mon[128973]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T19:01:43.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:43 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:43.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:43 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:43.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:43 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:43.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:43 smithi078 ceph-mon[127425]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T19:01:43.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:43 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:43.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:43 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:43.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:43 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:43.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:43 smithi102 ceph-mon[129280]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 57 KiB/s, 0 objects/s recovering 2023-07-14T19:01:43.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:43 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:43.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:43 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:43.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:43 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:45.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:45 smithi088 ceph-mon[128973]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:45.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:45 smithi078 ceph-mon[127425]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:45.622 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:45 smithi102 ceph-mon[129280]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:45.719 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.710+0000 7f722e7fc700 1 -- 172.21.15.102:0/4263405157 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f721c000bf0 con 0x7f72240a1150 2023-07-14T19:01:45.720 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 11 on host 'smithi102' 2023-07-14T19:01:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f72240a1150 msgr2=0x7f72240a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:45.720 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f72240a1150 0x7f72240a3600 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7238198b60 tx=0x7f7228009190 comp rx=0 tx=0).stop 2023-07-14T19:01:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238101850 msgr2=0x7f723819f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238101850 0x7f723819f230 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f723000b970 tx=0x7f723000bc80 comp rx=0 tx=0).stop 2023-07-14T19:01:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 shutdown_connections 2023-07-14T19:01:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f72240a1150 0x7f72240a3600 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:45.721 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f72380ffcc0 0x7f7238197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7238101850 0x7f723819f230 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 --2- 172.21.15.102:0/4263405157 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7238100ec0 0x7f72381981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 >> 172.21.15.102:0/4263405157 conn(0x7f72380fb2d0 msgr2=0x7f72380fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 shutdown_connections 2023-07-14T19:01:45.722 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:45.713+0000 7f723f6fb700 1 -- 172.21.15.102:0/4263405157 wait complete. 2023-07-14T19:01:46.150 DEBUG:teuthology.orchestra.run.smithi102:osd.11> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.11.service 2023-07-14T19:01:46.152 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2023-07-14T19:01:46.153 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T19:01:46.186 INFO:journalctl@ceph.osd.11.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T19:01:46.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:46.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:46.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:46 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:46.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.645 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:46.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:46.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.646 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:46.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:46.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:46.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:46.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:46 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:46.724 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:01:47.131 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:01:46 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:01:46.956+0000 7f41cbfeb640 -1 osd.11 0 log_to_monitors true 2023-07-14T19:01:47.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:47 smithi102 ceph-mon[129280]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:47.404 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:47 smithi102 ceph-mon[129280]: from='osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:01:47.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:47 smithi078 ceph-mon[127425]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:47.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:47 smithi078 ceph-mon[127425]: from='osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:01:47.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:47 smithi078 ceph-mon[127425]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:01:47.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:47 smithi088 ceph-mon[128973]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail; 48 KiB/s, 0 objects/s recovering 2023-07-14T19:01:47.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:47 smithi088 ceph-mon[128973]: from='osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:01:47.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:47 smithi088 ceph-mon[128973]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:01:47.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:47 smithi102 ceph-mon[129280]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2023-07-14T19:01:47.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 -- 172.21.15.78:0/3093664755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460100ec0 msgr2=0x7fd460101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:47.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 --2- 172.21.15.78:0/3093664755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460100ec0 0x7fd460101310 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd45c009990 tx=0x7fd45c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:01:47.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 -- 172.21.15.78:0/3093664755 shutdown_connections 2023-07-14T19:01:47.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 --2- 172.21.15.78:0/3093664755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd460101850 0x7fd460108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:47.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 --2- 172.21.15.78:0/3093664755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4600ffcc0 0x7fd4601000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:47.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 --2- 172.21.15.78:0/3093664755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460100ec0 0x7fd460101310 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:47.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 -- 172.21.15.78:0/3093664755 >> 172.21.15.78:0/3093664755 conn(0x7fd4600fb2d0 msgr2=0x7fd4600fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:47.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 -- 172.21.15.78:0/3093664755 shutdown_connections 2023-07-14T19:01:47.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.900+0000 7fd4681f4700 1 -- 172.21.15.78:0/3093664755 wait complete. 2023-07-14T19:01:47.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd4681f4700 1 Processor -- start 2023-07-14T19:01:47.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd4681f4700 1 -- start start 2023-07-14T19:01:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd4681f4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4600ffcc0 0x7fd460195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd4681f4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd460100ec0 0x7fd460195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd4681f4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460101850 0x7fd46019cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd4681f4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46019d4b0 con 0x7fd460101850 2023-07-14T19:01:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd4681f4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46019d5f0 con 0x7fd460100ec0 2023-07-14T19:01:47.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd4681f4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46019d780 con 0x7fd4600ffcc0 2023-07-14T19:01:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd466791700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460101850 0x7fd46019cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd466791700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460101850 0x7fd46019cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34070/0 (socket says 172.21.15.78:34070) 2023-07-14T19:01:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd466791700 1 -- 172.21.15.78:0/830472777 learned_addr learned my addr 172.21.15.78:0/830472777 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:01:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd465f90700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4600ffcc0 0x7fd460195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd46578f700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd460100ec0 0x7fd460195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd466791700 1 -- 172.21.15.78:0/830472777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4600ffcc0 msgr2=0x7fd460195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd466791700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4600ffcc0 0x7fd460195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:47.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd466791700 1 -- 172.21.15.78:0/830472777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd460100ec0 msgr2=0x7fd460195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:47.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd466791700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd460100ec0 0x7fd460195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:47.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.901+0000 7fd466791700 1 -- 172.21.15.78:0/830472777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45c0095f0 con 0x7fd460101850 2023-07-14T19:01:47.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.902+0000 7fd466791700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460101850 0x7fd46019cee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd45800b9a0 tx=0x7fd45800bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:47.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.902+0000 7fd456ffd700 1 -- 172.21.15.78:0/830472777 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd45800d850 con 0x7fd460101850 2023-07-14T19:01:47.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.902+0000 7fd456ffd700 1 -- 172.21.15.78:0/830472777 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd45800de40 con 0x7fd460101850 2023-07-14T19:01:47.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.902+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd46019da60 con 0x7fd460101850 2023-07-14T19:01:47.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.902+0000 7fd456ffd700 1 -- 172.21.15.78:0/830472777 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd458017400 con 0x7fd460101850 2023-07-14T19:01:47.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.902+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd46019df10 con 0x7fd460101850 2023-07-14T19:01:47.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.903+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd460107310 con 0x7fd460101850 2023-07-14T19:01:47.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.904+0000 7fd456ffd700 1 -- 172.21.15.78:0/830472777 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd45800d9b0 con 0x7fd460101850 2023-07-14T19:01:47.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.905+0000 7fd456ffd700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd44c0a1150 0x7fd44c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:47.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.905+0000 7fd465f90700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd44c0a1150 0x7fd44c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:47.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.905+0000 7fd456ffd700 1 -- 172.21.15.78:0/830472777 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 6831+0+0 (secure 0 0 0) 0x7fd458006e40 con 0x7fd460101850 2023-07-14T19:01:47.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.905+0000 7fd465f90700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd44c0a1150 0x7fd44c0a3600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd460100d20 tx=0x7fd45000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:47.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:47.908+0000 7fd456ffd700 1 -- 172.21.15.78:0/830472777 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd458095e10 con 0x7fd460101850 2023-07-14T19:01:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.093+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fd46002caf0 con 0x7fd460101850 2023-07-14T19:01:48.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.094+0000 7fd456ffd700 1 -- 172.21.15.78:0/830472777 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v74) v1 ==== 74+0+133 (secure 0 0 0) 0x7fd458099430 con 0x7fd460101850 2023-07-14T19:01:48.098 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:01:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd44c0a1150 msgr2=0x7fd44c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd44c0a1150 0x7fd44c0a3600 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd460100d20 tx=0x7fd45000b410 comp rx=0 tx=0).stop 2023-07-14T19:01:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460101850 msgr2=0x7fd46019cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460101850 0x7fd46019cee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd45800b9a0 tx=0x7fd45800bcb0 comp rx=0 tx=0).stop 2023-07-14T19:01:48.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 shutdown_connections 2023-07-14T19:01:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd44c0a1150 0x7fd44c0a3600 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd460101850 0x7fd46019cee0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4600ffcc0 0x7fd460195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 --2- 172.21.15.78:0/830472777 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd460100ec0 0x7fd460195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 >> 172.21.15.78:0/830472777 conn(0x7fd4600fb2d0 msgr2=0x7fd460104450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 shutdown_connections 2023-07-14T19:01:48.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:48.096+0000 7fd4681f4700 1 -- 172.21.15.78:0/830472777 wait complete. 2023-07-14T19:01:48.478 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":74,"num_osds":12,"num_up_osds":11,"osd_up_since":1689361288,"num_in_osds":12,"osd_in_since":1689361291,"num_remapped_pgs":0} 2023-07-14T19:01:48.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:01:48.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:01:48.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:48.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:01:48.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:01:48.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/830472777' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:01:48.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:48.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: Detected new or changed devices on smithi102 2023-07-14T19:01:48.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:48.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:48.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:48.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:48.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:48.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:48.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: Adjusting osd_memory_target on smithi102 to 4292M 2023-07-14T19:01:48.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:48.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:48.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:48.802 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:48 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.131 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:01:48.717+0000 7f41c1663700 -1 osd.11 0 waiting for initial osdmap 2023-07-14T19:01:49.131 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:01:48.728+0000 7f41b9451700 -1 osd.11 75 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-07-14T19:01:49.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:01:49.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:01:49.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:49.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:01:49.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:01:49.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/830472777' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:01:49.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:49.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: Detected new or changed devices on smithi102 2023-07-14T19:01:49.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:49.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:49.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:49.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:49.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: Adjusting osd_memory_target on smithi102 to 4292M 2023-07-14T19:01:49.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:49.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:49.134 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:48 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2023-07-14T19:01:49.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: osdmap e74: 12 total, 11 up, 12 in 2023-07-14T19:01:49.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:49.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:01:49.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2023-07-14T19:01:49.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/830472777' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:01:49.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:49.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: Detected new or changed devices on smithi102 2023-07-14T19:01:49.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:49.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:49.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:49.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2023-07-14T19:01:49.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: Adjusting osd_memory_target on smithi102 to 4292M 2023-07-14T19:01:49.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:49.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:49.215 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:48 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:49.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd stat -f json 2023-07-14T19:01:49.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:49 smithi078 ceph-mon[127425]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T19:01:49.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:49 smithi078 ceph-mon[127425]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:01:49.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:49 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:49.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:49 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:50.046 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:01:50.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:49 smithi102 ceph-mon[129280]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T19:01:50.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:49 smithi102 ceph-mon[129280]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:01:50.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:49 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:50.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:49 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:50.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:49 smithi088 ceph-mon[128973]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2023-07-14T19:01:50.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:49 smithi088 ceph-mon[128973]: osdmap e75: 12 total, 11 up, 12 in 2023-07-14T19:01:50.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:49 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:50.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:49 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:50.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:50 smithi078 ceph-mon[127425]: purged_snaps scrub starts 2023-07-14T19:01:50.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:50 smithi078 ceph-mon[127425]: purged_snaps scrub ok 2023-07-14T19:01:50.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:50 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:50.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:50 smithi078 ceph-mon[127425]: osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] boot 2023-07-14T19:01:50.837 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:50 smithi078 ceph-mon[127425]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:01:50.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:50 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:50.838 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:50 smithi078 ceph-mon[127425]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:51.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:50 smithi102 ceph-mon[129280]: purged_snaps scrub starts 2023-07-14T19:01:51.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:50 smithi102 ceph-mon[129280]: purged_snaps scrub ok 2023-07-14T19:01:51.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:50 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:51.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:50 smithi102 ceph-mon[129280]: osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] boot 2023-07-14T19:01:51.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:50 smithi102 ceph-mon[129280]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:01:51.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:50 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:51.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:50 smithi102 ceph-mon[129280]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:51.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:50 smithi088 ceph-mon[128973]: purged_snaps scrub starts 2023-07-14T19:01:51.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:50 smithi088 ceph-mon[128973]: purged_snaps scrub ok 2023-07-14T19:01:51.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:50 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:51.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:50 smithi088 ceph-mon[128973]: osd.11 [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] boot 2023-07-14T19:01:51.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:50 smithi088 ceph-mon[128973]: osdmap e76: 12 total, 12 up, 12 in 2023-07-14T19:01:51.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:50 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2023-07-14T19:01:51.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:50 smithi088 ceph-mon[128973]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 300 MiB used, 983 GiB / 983 GiB avail 2023-07-14T19:01:51.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 -- 172.21.15.78:0/923050894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49040ffdb0 msgr2=0x7f4904100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:51.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 --2- 172.21.15.78:0/923050894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49040ffdb0 0x7f4904100200 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f48f4009990 tx=0x7f48f4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 -- 172.21.15.78:0/923050894 shutdown_connections 2023-07-14T19:01:51.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 --2- 172.21.15.78:0/923050894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4904100740 0x7f4904107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 --2- 172.21.15.78:0/923050894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49040febb0 0x7f49040fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 --2- 172.21.15.78:0/923050894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f49040ffdb0 0x7f4904100200 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 -- 172.21.15.78:0/923050894 >> 172.21.15.78:0/923050894 conn(0x7f49040fa1e0 msgr2=0x7f49040fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:51.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 -- 172.21.15.78:0/923050894 shutdown_connections 2023-07-14T19:01:51.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.232+0000 7f49090cb700 1 -- 172.21.15.78:0/923050894 wait complete. 2023-07-14T19:01:51.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f49090cb700 1 Processor -- start 2023-07-14T19:01:51.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f49090cb700 1 -- start start 2023-07-14T19:01:51.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f49090cb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49040febb0 0x7f49041947d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:51.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f49090cb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49040ffdb0 0x7f4904194d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:51.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f49090cb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4904100740 0x7f490419bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:51.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f49090cb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490419c360 con 0x7f4904100740 2023-07-14T19:01:51.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f49090cb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490419c4a0 con 0x7f49040ffdb0 2023-07-14T19:01:51.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f49090cb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490419c630 con 0x7f49040febb0 2023-07-14T19:01:51.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f490359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4904100740 0x7f490419bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:51.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.233+0000 7f4902d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49040febb0 0x7f49041947d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:51.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f4902d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49040febb0 0x7f49041947d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.78:50438/0 (socket says 172.21.15.78:50438) 2023-07-14T19:01:51.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f4902d9d700 1 -- 172.21.15.78:0/2852871313 learned_addr learned my addr 172.21.15.78:0/2852871313 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:01:51.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f490259c700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49040ffdb0 0x7f4904194d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:51.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f490359e700 1 -- 172.21.15.78:0/2852871313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49040febb0 msgr2=0x7f49041947d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:51.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f490359e700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49040febb0 0x7f49041947d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f490359e700 1 -- 172.21.15.78:0/2852871313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49040ffdb0 msgr2=0x7f4904194d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:51.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f490359e700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49040ffdb0 0x7f4904194d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f490359e700 1 -- 172.21.15.78:0/2852871313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f40095f0 con 0x7f4904100740 2023-07-14T19:01:51.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f4902d9d700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49040febb0 0x7f49041947d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:01:51.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f490259c700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49040ffdb0 0x7f4904194d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:01:51.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f490359e700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4904100740 0x7f490419bd90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f48fc007f40 tx=0x7f48fc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:51.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f48fbfff700 1 -- 172.21.15.78:0/2852871313 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc004db0 con 0x7f4904100740 2023-07-14T19:01:51.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490419c910 con 0x7f4904100740 2023-07-14T19:01:51.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f48fbfff700 1 -- 172.21.15.78:0/2852871313 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f48fc00f040 con 0x7f4904100740 2023-07-14T19:01:51.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f48fbfff700 1 -- 172.21.15.78:0/2852871313 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc005950 con 0x7f4904100740 2023-07-14T19:01:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.234+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490419cdc0 con 0x7f4904100740 2023-07-14T19:01:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.235+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4904106220 con 0x7f4904100740 2023-07-14T19:01:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.236+0000 7f48fbfff700 1 -- 172.21.15.78:0/2852871313 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f48fc005000 con 0x7f4904100740 2023-07-14T19:01:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.237+0000 7f48fbfff700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f48f00a1150 0x7f48f00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:51.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.237+0000 7f4902d9d700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f48f00a1150 0x7f48f00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:51.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.237+0000 7f48fbfff700 1 -- 172.21.15.78:0/2852871313 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f48fc0cbe50 con 0x7f4904100740 2023-07-14T19:01:51.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.237+0000 7f4902d9d700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f48f00a1150 0x7f48f00a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0104e0 tx=0x7f48ec00b450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:51.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.241+0000 7f48fbfff700 1 -- 172.21.15.78:0/2852871313 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f48fc04e780 con 0x7f4904100740 2023-07-14T19:01:51.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.428+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f4904061250 con 0x7f4904100740 2023-07-14T19:01:51.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.429+0000 7f48fbfff700 1 -- 172.21.15.78:0/2852871313 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v77) v1 ==== 74+0+133 (secure 0 0 0) 0x7f48fc0959a0 con 0x7f4904100740 2023-07-14T19:01:51.434 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:01:51.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f48f00a1150 msgr2=0x7f48f00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f48f00a1150 0x7f48f00a3600 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f48ec0104e0 tx=0x7f48ec00b450 comp rx=0 tx=0).stop 2023-07-14T19:01:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4904100740 msgr2=0x7f490419bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:51.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4904100740 0x7f490419bd90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f48fc007f40 tx=0x7f48fc00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 shutdown_connections 2023-07-14T19:01:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f48f00a1150 0x7f48f00a3600 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4904100740 0x7f490419bd90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f49040febb0 0x7f49041947d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 --2- 172.21.15.78:0/2852871313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f49040ffdb0 0x7f4904194d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:51.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 >> 172.21.15.78:0/2852871313 conn(0x7f49040fa1e0 msgr2=0x7f4904103360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:51.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 shutdown_connections 2023-07-14T19:01:51.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:51.431+0000 7f49090cb700 1 -- 172.21.15.78:0/2852871313 wait complete. 2023-07-14T19:01:51.799 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":77,"num_osds":12,"num_up_osds":12,"osd_up_since":1689361309,"num_in_osds":12,"osd_in_since":1689361291,"num_remapped_pgs":0} 2023-07-14T19:01:51.800 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:01:52.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:51 smithi078 ceph-mon[127425]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:01:52.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:51 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2852871313' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:01:52.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:51 smithi102 ceph-mon[129280]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:01:52.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:51 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2852871313' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:01:52.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:51 smithi088 ceph-mon[128973]: osdmap e77: 12 total, 12 up, 12 in 2023-07-14T19:01:52.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:51 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2852871313' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-07-14T19:01:52.354 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:01:52.894 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:52 smithi078 ceph-mon[127425]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:53.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:52 smithi102 ceph-mon[129280]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:53.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:52 smithi088 ceph-mon[128973]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 327 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:53.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 -- 172.21.15.78:0/1794692201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 msgr2=0x7f1e2c0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:53.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/1794692201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 0x7f1e2c0ff210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1e200099c0 tx=0x7f1e20009cd0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 -- 172.21.15.78:0/1794692201 shutdown_connections 2023-07-14T19:01:53.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/1794692201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e2c103ee0 0x7f1e2c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/1794692201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 0x7f1e2c0ff210 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/1794692201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e2c0ff7e0 0x7f1e2c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 -- 172.21.15.78:0/1794692201 >> 172.21.15.78:0/1794692201 conn(0x7f1e2c075ca0 msgr2=0x7f1e2c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:53.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 -- 172.21.15.78:0/1794692201 shutdown_connections 2023-07-14T19:01:53.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.553+0000 7f1e31cbe700 1 -- 172.21.15.78:0/1794692201 wait complete. 2023-07-14T19:01:53.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e31cbe700 1 Processor -- start 2023-07-14T19:01:53.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e31cbe700 1 -- start start 2023-07-14T19:01:53.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e31cbe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 0x7f1e2c19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:53.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e31cbe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e2c0ff7e0 0x7f1e2c19af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:53.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e31cbe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e2c103ee0 0x7f1e2c194a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:53.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e31cbe700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c1950b0 con 0x7f1e2c0fee40 2023-07-14T19:01:53.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e31cbe700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c195220 con 0x7f1e2c0ff7e0 2023-07-14T19:01:53.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e31cbe700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2c1953e0 con 0x7f1e2c103ee0 2023-07-14T19:01:53.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 0x7f1e2c19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:53.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 0x7f1e2c19a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34100/0 (socket says 172.21.15.78:34100) 2023-07-14T19:01:53.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2b7fe700 1 -- 172.21.15.78:0/3445128888 learned_addr learned my addr 172.21.15.78:0/3445128888 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:01:53.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2affd700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e2c0ff7e0 0x7f1e2c19af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:53.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2bfff700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e2c103ee0 0x7f1e2c194a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:53.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2b7fe700 1 -- 172.21.15.78:0/3445128888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e2c103ee0 msgr2=0x7f1e2c194a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:53.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2b7fe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e2c103ee0 0x7f1e2c194a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2b7fe700 1 -- 172.21.15.78:0/3445128888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e2c0ff7e0 msgr2=0x7f1e2c19af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:53.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2b7fe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e2c0ff7e0 0x7f1e2c19af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2b7fe700 1 -- 172.21.15.78:0/3445128888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e200095f0 con 0x7f1e2c0fee40 2023-07-14T19:01:53.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.554+0000 7f1e2affd700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e2c0ff7e0 0x7f1e2c19af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:01:53.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.555+0000 7f1e2b7fe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 0x7f1e2c19a9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1e20009990 tx=0x7f1e200038d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:53.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.555+0000 7f1e28ff9700 1 -- 172.21.15.78:0/3445128888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e2001d070 con 0x7f1e2c0fee40 2023-07-14T19:01:53.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.555+0000 7f1e28ff9700 1 -- 172.21.15.78:0/3445128888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1e20004210 con 0x7f1e2c0fee40 2023-07-14T19:01:53.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.555+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c195660 con 0x7f1e2c0fee40 2023-07-14T19:01:53.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.555+0000 7f1e28ff9700 1 -- 172.21.15.78:0/3445128888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e20022e30 con 0x7f1e2c0fee40 2023-07-14T19:01:53.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.555+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e2c1a19b0 con 0x7f1e2c0fee40 2023-07-14T19:01:53.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.556+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2c04e340 con 0x7f1e2c0fee40 2023-07-14T19:01:53.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.557+0000 7f1e28ff9700 1 -- 172.21.15.78:0/3445128888 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1e20022600 con 0x7f1e2c0fee40 2023-07-14T19:01:53.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.557+0000 7f1e28ff9700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e140a1250 0x7f1e140a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:53.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.557+0000 7f1e28ff9700 1 -- 172.21.15.78:0/3445128888 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1e200cd810 con 0x7f1e2c0fee40 2023-07-14T19:01:53.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.557+0000 7f1e2affd700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e140a1250 0x7f1e140a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:53.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.558+0000 7f1e2affd700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e140a1250 0x7f1e140a3700 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c077370 tx=0x7f1e1c005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:53.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.561+0000 7f1e28ff9700 1 -- 172.21.15.78:0/3445128888 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1e200973f0 con 0x7f1e2c0fee40 2023-07-14T19:01:53.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.747+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1e2c195f50 con 0x7f1e2c0fee40 2023-07-14T19:01:53.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.748+0000 7f1e28ff9700 1 -- 172.21.15.78:0/3445128888 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18384 (secure 0 0 0) 0x7f1e2009aa10 con 0x7f1e2c0fee40 2023-07-14T19:01:53.752 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:01:53.753 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":77,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:01:50.722377+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:01:53.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e140a1250 msgr2=0x7f1e140a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:53.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e140a1250 0x7f1e140a3700 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1e2c077370 tx=0x7f1e1c005f50 comp rx=0 tx=0).stop 2023-07-14T19:01:53.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 msgr2=0x7f1e2c19a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:53.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 0x7f1e2c19a9c0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f1e20009990 tx=0x7f1e200038d0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 shutdown_connections 2023-07-14T19:01:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1e140a1250 0x7f1e140a3700 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e2c103ee0 0x7f1e2c194a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1e2c0fee40 0x7f1e2c19a9c0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 --2- 172.21.15.78:0/3445128888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1e2c0ff7e0 0x7f1e2c19af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 >> 172.21.15.78:0/3445128888 conn(0x7f1e2c075ca0 msgr2=0x7f1e2c0fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 shutdown_connections 2023-07-14T19:01:53.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:53.750+0000 7f1e31cbe700 1 -- 172.21.15.78:0/3445128888 wait complete. 2023-07-14T19:01:54.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:53 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3445128888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:01:54.107 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:01:54.107 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd pool get .mgr pg_num 2023-07-14T19:01:54.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:53 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3445128888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:01:54.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:53 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3445128888' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:01:54.656 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:01:55.100 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:54 smithi078 ceph-mon[127425]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:55.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:54 smithi102 ceph-mon[129280]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:55.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:54 smithi088 ceph-mon[128973]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:55.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 -- 172.21.15.78:0/1556823019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d00fee40 msgr2=0x7f52d00ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:55.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 --2- 172.21.15.78:0/1556823019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d00fee40 0x7f52d00ff210 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f52cc0099c0 tx=0x7f52cc009cd0 comp rx=0 tx=0).stop 2023-07-14T19:01:55.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 -- 172.21.15.78:0/1556823019 shutdown_connections 2023-07-14T19:01:55.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 --2- 172.21.15.78:0/1556823019 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d0103ee0 0x7f52d010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:55.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 --2- 172.21.15.78:0/1556823019 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d00fee40 0x7f52d00ff210 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:55.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 --2- 172.21.15.78:0/1556823019 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52d00ff7e0 0x7f52d01037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:55.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 -- 172.21.15.78:0/1556823019 >> 172.21.15.78:0/1556823019 conn(0x7f52d0075ca0 msgr2=0x7f52d00760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:55.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 -- 172.21.15.78:0/1556823019 shutdown_connections 2023-07-14T19:01:55.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 -- 172.21.15.78:0/1556823019 wait complete. 2023-07-14T19:01:55.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 Processor -- start 2023-07-14T19:01:55.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.960+0000 7f52d7822700 1 -- start start 2023-07-14T19:01:55.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d7822700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d00fee40 0x7f52d0195cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:55.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d7822700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52d00ff7e0 0x7f52d0196210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:55.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d7822700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d0103ee0 0x7f52d01995d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:55.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d7822700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0199bd0 con 0x7f52d0103ee0 2023-07-14T19:01:55.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d7822700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0199d40 con 0x7f52d00fee40 2023-07-14T19:01:55.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d7822700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52d0199f00 con 0x7f52d00ff7e0 2023-07-14T19:01:55.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d0103ee0 0x7f52d01995d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:55.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d4dbd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52d00ff7e0 0x7f52d0196210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:55.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d0103ee0 0x7f52d01995d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34116/0 (socket says 172.21.15.78:34116) 2023-07-14T19:01:55.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 -- 172.21.15.78:0/2778124059 learned_addr learned my addr 172.21.15.78:0/2778124059 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:01:55.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d55be700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d00fee40 0x7f52d0195cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 -- 172.21.15.78:0/2778124059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52d00ff7e0 msgr2=0x7f52d0196210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52d00ff7e0 0x7f52d0196210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 -- 172.21.15.78:0/2778124059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d00fee40 msgr2=0x7f52d0195cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d00fee40 0x7f52d0195cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 -- 172.21.15.78:0/2778124059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc0095f0 con 0x7f52d0103ee0 2023-07-14T19:01:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d5dbf700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d0103ee0 0x7f52d01995d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f52c800ab50 tx=0x7f52c800ae60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52c67fc700 1 -- 172.21.15.78:0/2778124059 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c800f070 con 0x7f52d0103ee0 2023-07-14T19:01:55.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52c67fc700 1 -- 172.21.15.78:0/2778124059 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f52c8007a80 con 0x7f52d0103ee0 2023-07-14T19:01:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52c67fc700 1 -- 172.21.15.78:0/2778124059 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52c8005280 con 0x7f52d0103ee0 2023-07-14T19:01:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52d019a1e0 con 0x7f52d0103ee0 2023-07-14T19:01:55.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.961+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52d01a2b30 con 0x7f52d0103ee0 2023-07-14T19:01:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.962+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52d004e340 con 0x7f52d0103ee0 2023-07-14T19:01:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.963+0000 7f52c67fc700 1 -- 172.21.15.78:0/2778124059 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f52c8004ad0 con 0x7f52d0103ee0 2023-07-14T19:01:55.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.964+0000 7f52c67fc700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f52bc0a1150 0x7f52bc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:55.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.964+0000 7f52d55be700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f52bc0a1150 0x7f52bc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:55.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.964+0000 7f52c67fc700 1 -- 172.21.15.78:0/2778124059 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f52c80cb8d0 con 0x7f52d0103ee0 2023-07-14T19:01:55.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.965+0000 7f52d55be700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f52bc0a1150 0x7f52bc0a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009fd0 tx=0x7f52cc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:55.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:55.969+0000 7f52c67fc700 1 -- 172.21.15.78:0/2778124059 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f52c8095550 con 0x7f52d0103ee0 2023-07-14T19:01:56.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.156+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f52d01a2dc0 con 0x7f52d0103ee0 2023-07-14T19:01:56.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.156+0000 7f52c67fc700 1 -- 172.21.15.78:0/2778124059 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f52c8098b70 con 0x7f52d0103ee0 2023-07-14T19:01:56.161 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:01:56.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f52bc0a1150 msgr2=0x7f52bc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:56.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f52bc0a1150 0x7f52bc0a3600 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f52cc009fd0 tx=0x7f52cc005fb0 comp rx=0 tx=0).stop 2023-07-14T19:01:56.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d0103ee0 msgr2=0x7f52d01995d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:56.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d0103ee0 0x7f52d01995d0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f52c800ab50 tx=0x7f52c800ae60 comp rx=0 tx=0).stop 2023-07-14T19:01:56.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 shutdown_connections 2023-07-14T19:01:56.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f52bc0a1150 0x7f52bc0a3600 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:56.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f52d00fee40 0x7f52d0195cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:56.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f52d0103ee0 0x7f52d01995d0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 --2- 172.21.15.78:0/2778124059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f52d00ff7e0 0x7f52d0196210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 >> 172.21.15.78:0/2778124059 conn(0x7f52d0075ca0 msgr2=0x7f52d0102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 shutdown_connections 2023-07-14T19:01:56.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:56.158+0000 7f52d7822700 1 -- 172.21.15.78:0/2778124059 wait complete. 2023-07-14T19:01:56.509 INFO:tasks.cephadm:Adding mds.a on smithi078 2023-07-14T19:01:56.509 INFO:tasks.cephadm:Adding mds.d on smithi078 2023-07-14T19:01:56.509 INFO:tasks.cephadm:Adding mds.g on smithi078 2023-07-14T19:01:56.509 INFO:tasks.cephadm:Adding mds.j on smithi078 2023-07-14T19:01:56.510 INFO:tasks.cephadm:Adding mds.b on smithi088 2023-07-14T19:01:56.510 INFO:tasks.cephadm:Adding mds.e on smithi088 2023-07-14T19:01:56.510 INFO:tasks.cephadm:Adding mds.h on smithi088 2023-07-14T19:01:56.510 INFO:tasks.cephadm:Adding mds.k on smithi088 2023-07-14T19:01:56.510 INFO:tasks.cephadm:Adding mds.c on smithi102 2023-07-14T19:01:56.511 INFO:tasks.cephadm:Adding mds.f on smithi102 2023-07-14T19:01:56.511 INFO:tasks.cephadm:Adding mds.i on smithi102 2023-07-14T19:01:56.511 INFO:tasks.cephadm:Adding mds.l on smithi102 2023-07-14T19:01:56.511 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph orch apply mds all '12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l' 2023-07-14T19:01:56.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:56 smithi078 ceph-mon[127425]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:56.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:56 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2778124059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:01:56.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:56 smithi102 ceph-mon[129280]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:56.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:56 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2778124059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:01:56.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:56 smithi088 ceph-mon[128973]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:56.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:56 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2778124059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:01:57.089 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.c/config 2023-07-14T19:01:58.266 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 -- 172.21.15.102:0/953743953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80febb0 msgr2=0x7f4cf80fefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:58.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 --2- 172.21.15.102:0/953743953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80febb0 0x7f4cf80fefc0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4009990 tx=0x7f4cf4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 -- 172.21.15.102:0/953743953 shutdown_connections 2023-07-14T19:01:58.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 --2- 172.21.15.102:0/953743953 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cf8100740 0x7f4cf8107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.267 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 --2- 172.21.15.102:0/953743953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80febb0 0x7f4cf80fefc0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 --2- 172.21.15.102:0/953743953 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf80ffdb0 0x7f4cf8100200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 -- 172.21.15.102:0/953743953 >> 172.21.15.102:0/953743953 conn(0x7f4cf80fa1e0 msgr2=0x7f4cf80fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:58.268 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 -- 172.21.15.102:0/953743953 shutdown_connections 2023-07-14T19:01:58.269 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 -- 172.21.15.102:0/953743953 wait complete. 2023-07-14T19:01:58.272 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.262+0000 7f4d0035e700 1 Processor -- start 2023-07-14T19:01:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4d0035e700 1 -- start start 2023-07-14T19:01:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4d0035e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cf80febb0 0x7f4cf8068cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4d0035e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80ffdb0 0x7f4cf819a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4d0035e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf8100740 0x7f4cf819cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:58.273 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4d0035e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf819d510 con 0x7f4cf80febb0 2023-07-14T19:01:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4d0035e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf819d680 con 0x7f4cf8100740 2023-07-14T19:01:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4d0035e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf819d840 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80ffdb0 0x7f4cf819a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80ffdb0 0x7f4cf819a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:52532/0 (socket says 172.21.15.102:52532) 2023-07-14T19:01:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 -- 172.21.15.102:0/2930236499 learned_addr learned my addr 172.21.15.102:0/2930236499 (peer_addr_for_me v2:172.21.15.102:0/0) 2023-07-14T19:01:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfe8fb700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf8100740 0x7f4cf819cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:58.274 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfe0fa700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cf80febb0 0x7f4cf8068cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:58.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 -- 172.21.15.102:0/2930236499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf8100740 msgr2=0x7f4cf819cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:58.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf8100740 0x7f4cf819cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 -- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cf80febb0 msgr2=0x7f4cf8068cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:58.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cf80febb0 0x7f4cf8068cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 -- 172.21.15.102:0/2930236499 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf40095f0 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.263+0000 7f4cfd8f9700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80ffdb0 0x7f4cf819a6e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800b530 tx=0x7f4ce800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:58.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.264+0000 7f4cef7fe700 1 -- 172.21.15.102:0/2930236499 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce8011830 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.275 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.264+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf819db20 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.264+0000 7f4cef7fe700 1 -- 172.21.15.102:0/2930236499 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4ce8011e20 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.264+0000 7f4cef7fe700 1 -- 172.21.15.102:0/2930236499 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce800f2e0 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.264+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf819e020 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.265+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf810a2f0 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.266+0000 7f4cef7fe700 1 -- 172.21.15.102:0/2930236499 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4ce8011990 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.266+0000 7f4cef7fe700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ce40a1150 0x7f4ce40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:58.276 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.266+0000 7f4cef7fe700 1 -- 172.21.15.102:0/2930236499 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4ce80cbfc0 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.267+0000 7f4cfe0fa700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ce40a1150 0x7f4ce40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:58.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.267+0000 7f4cfe0fa700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ce40a1150 0x7f4ce40a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4cf400b5c0 tx=0x7f4cf4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:58.277 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.271+0000 7f4cef7fe700 1 -- 172.21.15.102:0/2930236499 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4ce80fe780 con 0x7f4cf80ffdb0 2023-07-14T19:01:58.470 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.466+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f4cf8060b80 con 0x7f4ce40a1150 2023-07-14T19:01:58.478 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.474+0000 7f4cef7fe700 1 -- 172.21.15.102:0/2930236499 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f4cf8060b80 con 0x7f4ce40a1150 2023-07-14T19:01:58.478 INFO:teuthology.orchestra.run.smithi102.stdout:Scheduled mds.all update... 2023-07-14T19:01:58.480 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ce40a1150 msgr2=0x7f4ce40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ce40a1150 0x7f4ce40a3600 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4cf400b5c0 tx=0x7f4cf4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80ffdb0 msgr2=0x7f4cf819a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80ffdb0 0x7f4cf819a6e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800b530 tx=0x7f4ce800b840 comp rx=0 tx=0).stop 2023-07-14T19:01:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 shutdown_connections 2023-07-14T19:01:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ce40a1150 0x7f4ce40a3600 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4cf8100740 0x7f4cf819cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.481 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4cf80febb0 0x7f4cf8068cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 --2- 172.21.15.102:0/2930236499 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4cf80ffdb0 0x7f4cf819a6e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.476+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 >> 172.21.15.102:0/2930236499 conn(0x7f4cf80fa1e0 msgr2=0x7f4cf80fc510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.477+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 shutdown_connections 2023-07-14T19:01:58.482 INFO:teuthology.orchestra.run.smithi102.stderr:2023-07-14T19:01:58.477+0000 7f4d0035e700 1 -- 172.21.15.102:0/2930236499 wait complete. 2023-07-14T19:01:58.842 DEBUG:teuthology.orchestra.run.smithi078:mds.a> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.a.service 2023-07-14T19:01:58.845 DEBUG:teuthology.orchestra.run.smithi078:mds.d> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.d.service 2023-07-14T19:01:58.882 INFO:journalctl@ceph.mds.a.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T19:01:58.884 DEBUG:teuthology.orchestra.run.smithi078:mds.g> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.g.service 2023-07-14T19:01:58.916 INFO:journalctl@ceph.mds.d.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T19:01:58.917 DEBUG:teuthology.orchestra.run.smithi078:mds.j> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.j.service 2023-07-14T19:01:58.948 INFO:journalctl@ceph.mds.g.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T19:01:58.949 DEBUG:teuthology.orchestra.run.smithi088:mds.b> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.b.service 2023-07-14T19:01:58.952 DEBUG:teuthology.orchestra.run.smithi088:mds.e> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.e.service 2023-07-14T19:01:58.978 INFO:journalctl@ceph.mds.j.smithi078.stdout:-- Logs begin at Fri 2023-07-14 18:34:07 UTC. -- 2023-07-14T19:01:58.990 INFO:journalctl@ceph.mds.b.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T19:01:58.991 DEBUG:teuthology.orchestra.run.smithi088:mds.h> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.h.service 2023-07-14T19:01:59.020 INFO:journalctl@ceph.mds.e.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T19:01:59.022 DEBUG:teuthology.orchestra.run.smithi088:mds.k> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.k.service 2023-07-14T19:01:59.050 INFO:journalctl@ceph.mds.h.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T19:01:59.051 DEBUG:teuthology.orchestra.run.smithi102:mds.c> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.c.service 2023-07-14T19:01:59.053 DEBUG:teuthology.orchestra.run.smithi102:mds.f> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.f.service 2023-07-14T19:01:59.076 INFO:journalctl@ceph.mds.k.smithi088.stdout:-- Logs begin at Fri 2023-07-14 18:34:48 UTC. -- 2023-07-14T19:01:59.082 INFO:journalctl@ceph.mds.c.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T19:01:59.082 DEBUG:teuthology.orchestra.run.smithi102:mds.i> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.i.service 2023-07-14T19:01:59.108 INFO:journalctl@ceph.mds.f.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T19:01:59.109 DEBUG:teuthology.orchestra.run.smithi102:mds.l> sudo journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.l.service 2023-07-14T19:01:59.135 INFO:journalctl@ceph.mds.i.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T19:01:59.136 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2023-07-14T19:01:59.137 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:01:59.163 INFO:journalctl@ceph.mds.l.smithi102.stdout:-- Logs begin at Fri 2023-07-14 18:34:15 UTC. -- 2023-07-14T19:01:59.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 -- 172.21.15.78:0/3712214220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a441006a0 msgr2=0x7f8a44106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.289 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 --2- 172.21.15.78:0/3712214220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a441006a0 0x7f8a44106f60 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c00b0a0 tx=0x7f8a3c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 -- 172.21.15.78:0/3712214220 shutdown_connections 2023-07-14T19:01:59.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 --2- 172.21.15.78:0/3712214220 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a440feb20 0x7f8a440fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 --2- 172.21.15.78:0/3712214220 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a441006a0 0x7f8a44106f60 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 --2- 172.21.15.78:0/3712214220 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a440ffd10 0x7f8a44100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 -- 172.21.15.78:0/3712214220 >> 172.21.15.78:0/3712214220 conn(0x7f8a440fa100 msgr2=0x7f8a440fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:59.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 -- 172.21.15.78:0/3712214220 shutdown_connections 2023-07-14T19:01:59.290 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.285+0000 7f8a48e78700 1 -- 172.21.15.78:0/3712214220 wait complete. 2023-07-14T19:01:59.292 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a48e78700 1 Processor -- start 2023-07-14T19:01:59.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a48e78700 1 -- start start 2023-07-14T19:01:59.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a48e78700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a440feb20 0x7f8a44069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a48e78700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a440ffd10 0x7f8a4419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a48e78700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a441006a0 0x7f8a4419ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a48e78700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4419f560 con 0x7f8a440ffd10 2023-07-14T19:01:59.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a48e78700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4419f6d0 con 0x7f8a440feb20 2023-07-14T19:01:59.293 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a48e78700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4419f890 con 0x7f8a441006a0 2023-07-14T19:01:59.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a440ffd10 0x7f8a4419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a440ffd10 0x7f8a4419c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34126/0 (socket says 172.21.15.78:34126) 2023-07-14T19:01:59.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a41d9b700 1 -- 172.21.15.78:0/1011135600 learned_addr learned my addr 172.21.15.78:0/1011135600 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:01:59.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a4259c700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a440feb20 0x7f8a44069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a42d9d700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a441006a0 0x7f8a4419ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.294 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a41d9b700 1 -- 172.21.15.78:0/1011135600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a441006a0 msgr2=0x7f8a4419ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a41d9b700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a441006a0 0x7f8a4419ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a41d9b700 1 -- 172.21.15.78:0/1011135600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a440feb20 msgr2=0x7f8a44069360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.295 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a41d9b700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a440feb20 0x7f8a44069360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a41d9b700 1 -- 172.21.15.78:0/1011135600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c009d00 con 0x7f8a440ffd10 2023-07-14T19:01:59.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.286+0000 7f8a42d9d700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a441006a0 0x7f8a4419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:01:59.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.287+0000 7f8a41d9b700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a440ffd10 0x7f8a4419c930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800e9a0 tx=0x7f8a3800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:59.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.287+0000 7f8a337fe700 1 -- 172.21.15.78:0/1011135600 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a38009790 con 0x7f8a440ffd10 2023-07-14T19:01:59.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.287+0000 7f8a337fe700 1 -- 172.21.15.78:0/1011135600 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8a3800cbd0 con 0x7f8a440ffd10 2023-07-14T19:01:59.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.287+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4419fb70 con 0x7f8a440ffd10 2023-07-14T19:01:59.296 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.287+0000 7f8a337fe700 1 -- 172.21.15.78:0/1011135600 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a38018680 con 0x7f8a440ffd10 2023-07-14T19:01:59.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.287+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a441a00d0 con 0x7f8a440ffd10 2023-07-14T19:01:59.297 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.288+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a44105fd0 con 0x7f8a440ffd10 2023-07-14T19:01:59.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.289+0000 7f8a337fe700 1 -- 172.21.15.78:0/1011135600 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8a3800cd40 con 0x7f8a440ffd10 2023-07-14T19:01:59.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.289+0000 7f8a337fe700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a2c0a1220 0x7f8a2c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.289+0000 7f8a4259c700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a2c0a1220 0x7f8a2c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.289+0000 7f8a337fe700 1 -- 172.21.15.78:0/1011135600 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8a38014070 con 0x7f8a440ffd10 2023-07-14T19:01:59.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.290+0000 7f8a4259c700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a2c0a1220 0x7f8a2c0a36d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a34005fd0 tx=0x7f8a34005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:59.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.293+0000 7f8a337fe700 1 -- 172.21.15.78:0/1011135600 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a380964a0 con 0x7f8a440ffd10 2023-07-14T19:01:59.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.483+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a4402c8e0 con 0x7f8a440ffd10 2023-07-14T19:01:59.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.484+0000 7f8a337fe700 1 -- 172.21.15.78:0/1011135600 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18384 (secure 0 0 0) 0x7f8a38099ac0 con 0x7f8a440ffd10 2023-07-14T19:01:59.489 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:01:59.489 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":77,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:01:50.722377+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:01:59.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.485+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a2c0a1220 msgr2=0x7f8a2c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.485+0000 7f8a48e78700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a2c0a1220 0x7f8a2c0a36d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8a34005fd0 tx=0x7f8a34005e50 comp rx=0 tx=0).stop 2023-07-14T19:01:59.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.485+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a440ffd10 msgr2=0x7f8a4419c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a440ffd10 0x7f8a4419c930 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8a3800e9a0 tx=0x7f8a3800ed60 comp rx=0 tx=0).stop 2023-07-14T19:01:59.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 shutdown_connections 2023-07-14T19:01:59.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a2c0a1220 0x7f8a2c0a36d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a440feb20 0x7f8a44069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a441006a0 0x7f8a4419ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 --2- 172.21.15.78:0/1011135600 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a440ffd10 0x7f8a4419c930 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 >> 172.21.15.78:0/1011135600 conn(0x7f8a440fa100 msgr2=0x7f8a44103110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:59.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 shutdown_connections 2023-07-14T19:01:59.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.486+0000 7f8a48e78700 1 -- 172.21.15.78:0/1011135600 wait complete. 2023-07-14T19:01:59.502 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:01:59.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:01:59.628 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 -- 172.21.15.78:0/1916739427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008102ff0 msgr2=0x7f70081053d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 --2- 172.21.15.78:0/1916739427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008102ff0 0x7f70081053d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6ff8009990 tx=0x7f6ff8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 -- 172.21.15.78:0/1916739427 shutdown_connections 2023-07-14T19:01:59.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 --2- 172.21.15.78:0/1916739427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f70081006d0 0x7f7008102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 --2- 172.21.15.78:0/1916739427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7008105910 0x7f7008107d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 --2- 172.21.15.78:0/1916739427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008102ff0 0x7f70081053d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.629 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 -- 172.21.15.78:0/1916739427 >> 172.21.15.78:0/1916739427 conn(0x7f70080fa150 msgr2=0x7f70080fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:59.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 -- 172.21.15.78:0/1916739427 shutdown_connections 2023-07-14T19:01:59.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 -- 172.21.15.78:0/1916739427 wait complete. 2023-07-14T19:01:59.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.623+0000 7f700d168700 1 Processor -- start 2023-07-14T19:01:59.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700d168700 1 -- start start 2023-07-14T19:01:59.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700d168700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70081006d0 0x7f7008100040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700d168700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7008102ff0 0x7f70080fe680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700d168700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008105910 0x7f70080febc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.630 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700d168700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70080ff360 con 0x7f7008105910 2023-07-14T19:01:59.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700d168700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70080ff4d0 con 0x7f7008102ff0 2023-07-14T19:01:59.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700d168700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f700819f8b0 con 0x7f70081006d0 2023-07-14T19:01:59.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008105910 0x7f70080febc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008105910 0x7f70080febc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34142/0 (socket says 172.21.15.78:34142) 2023-07-14T19:01:59.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 -- 172.21.15.78:0/27962644 learned_addr learned my addr 172.21.15.78:0/27962644 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:01:59.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f7006d9d700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70081006d0 0x7f7008100040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700659c700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7008102ff0 0x7f70080fe680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 -- 172.21.15.78:0/27962644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70081006d0 msgr2=0x7f7008100040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70081006d0 0x7f7008100040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 -- 172.21.15.78:0/27962644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7008102ff0 msgr2=0x7f70080fe680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7008102ff0 0x7f70080fe680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 -- 172.21.15.78:0/27962644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80095f0 con 0x7f7008105910 2023-07-14T19:01:59.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700659c700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7008102ff0 0x7f70080fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:01:59.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700759e700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008105910 0x7f70080febc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f700000ada0 tx=0x7f700000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:59.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f6ffffff700 1 -- 172.21.15.78:0/27962644 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700000f070 con 0x7f7008105910 2023-07-14T19:01:59.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.624+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f700819fb90 con 0x7f7008105910 2023-07-14T19:01:59.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.625+0000 7f6ffffff700 1 -- 172.21.15.78:0/27962644 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7000007b30 con 0x7f7008105910 2023-07-14T19:01:59.633 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.625+0000 7f6ffffff700 1 -- 172.21.15.78:0/27962644 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000013ba0 con 0x7f7008105910 2023-07-14T19:01:59.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.625+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70081a00f0 con 0x7f7008105910 2023-07-14T19:01:59.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.625+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70080fbcc0 con 0x7f7008105910 2023-07-14T19:01:59.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.626+0000 7f6ffffff700 1 -- 172.21.15.78:0/27962644 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f700000c980 con 0x7f7008105910 2023-07-14T19:01:59.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.627+0000 7f6ffffff700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ff40a1150 0x7f6ff40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.627+0000 7f7006d9d700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ff40a1150 0x7f6ff40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.627+0000 7f6ffffff700 1 -- 172.21.15.78:0/27962644 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f70000cc8e0 con 0x7f7008105910 2023-07-14T19:01:59.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.627+0000 7f7006d9d700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ff40a1150 0x7f6ff40a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0005ea0 tx=0x7f6ff0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:59.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.630+0000 7f6ffffff700 1 -- 172.21.15.78:0/27962644 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f70000964c0 con 0x7f7008105910 2023-07-14T19:01:59.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:59.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:59.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l;count:12 2023-07-14T19:01:59.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:59.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:59.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:59.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:59.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:59.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:01:59.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:01:59.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:01:59 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:59.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.821+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f700802cd50 con 0x7f7008105910 2023-07-14T19:01:59.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.821+0000 7f6ffffff700 1 -- 172.21.15.78:0/27962644 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7000099ae0 con 0x7f7008105910 2023-07-14T19:01:59.825 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:01:59.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ff40a1150 msgr2=0x7f6ff40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ff40a1150 0x7f6ff40a3600 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6ff0005ea0 tx=0x7f6ff0005e30 comp rx=0 tx=0).stop 2023-07-14T19:01:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008105910 msgr2=0x7f70080febc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008105910 0x7f70080febc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f700000ada0 tx=0x7f700000c330 comp rx=0 tx=0).stop 2023-07-14T19:01:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 shutdown_connections 2023-07-14T19:01:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6ff40a1150 0x7f6ff40a3600 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7008105910 0x7f70080febc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70081006d0 0x7f7008100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 --2- 172.21.15.78:0/27962644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7008102ff0 0x7f70080fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 >> 172.21.15.78:0/27962644 conn(0x7f70080fa150 msgr2=0x7f70081013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 shutdown_connections 2023-07-14T19:01:59.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.823+0000 7f700d168700 1 -- 172.21.15.78:0/27962644 wait complete. 2023-07-14T19:01:59.839 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:01:59.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:59.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:59.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l;count:12 2023-07-14T19:01:59.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:59.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:01:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:01:59.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:01:59 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:59.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 -- 172.21.15.78:0/546172184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e00feb30 msgr2=0x7f84e00fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/546172184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e00feb30 0x7f84e00fef40 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f84d0009990 tx=0x7f84d0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 -- 172.21.15.78:0/546172184 shutdown_connections 2023-07-14T19:01:59.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/546172184 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e01006b0 0x7f84e0106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/546172184 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e00feb30 0x7f84e00fef40 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/546172184 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00ffd20 0x7f84e0100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 -- 172.21.15.78:0/546172184 >> 172.21.15.78:0/546172184 conn(0x7f84e00fa150 msgr2=0x7f84e00fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:01:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 -- 172.21.15.78:0/546172184 shutdown_connections 2023-07-14T19:01:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.956+0000 7f84e5b2f700 1 -- 172.21.15.78:0/546172184 wait complete. 2023-07-14T19:01:59.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84e5b2f700 1 Processor -- start 2023-07-14T19:01:59.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:01:59.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='client.24400 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:01:59.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: Saving service mds.all spec with placement smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l;count:12 2023-07-14T19:01:59.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:59.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:01:59.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:59.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:01:59.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:01:59.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:01:59.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:01:59.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:01:59 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:01:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84e5b2f700 1 -- start start 2023-07-14T19:01:59.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84e5b2f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00feb30 0x7f84e0069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84e5b2f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e00ffd20 0x7f84e019c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84e5b2f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e01006b0 0x7f84e019eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84e5b2f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e019f590 con 0x7f84e01006b0 2023-07-14T19:01:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84e5b2f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e019f700 con 0x7f84e00ffd20 2023-07-14T19:01:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84e5b2f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e019f8c0 con 0x7f84e00feb30 2023-07-14T19:01:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84dffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e01006b0 0x7f84e019eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84df7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00feb30 0x7f84e0069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84df7fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00feb30 0x7f84e0069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.78:50510/0 (socket says 172.21.15.78:50510) 2023-07-14T19:01:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84df7fe700 1 -- 172.21.15.78:0/3192901154 learned_addr learned my addr 172.21.15.78:0/3192901154 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:01:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.957+0000 7f84deffd700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e00ffd20 0x7f84e019c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dffff700 1 -- 172.21.15.78:0/3192901154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00feb30 msgr2=0x7f84e0069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dffff700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00feb30 0x7f84e0069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dffff700 1 -- 172.21.15.78:0/3192901154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e00ffd20 msgr2=0x7f84e019c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:01:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dffff700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e00ffd20 0x7f84e019c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:01:59.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dffff700 1 -- 172.21.15.78:0/3192901154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d00095f0 con 0x7f84e01006b0 2023-07-14T19:01:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84df7fe700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00feb30 0x7f84e0069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:01:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dffff700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e01006b0 0x7f84e019eee0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f84d8007f10 tx=0x7f84d800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dcff9700 1 -- 172.21.15.78:0/3192901154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8004db0 con 0x7f84e01006b0 2023-07-14T19:01:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dcff9700 1 -- 172.21.15.78:0/3192901154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f84d800f040 con 0x7f84e01006b0 2023-07-14T19:01:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84dcff9700 1 -- 172.21.15.78:0/3192901154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84d8005960 con 0x7f84e01006b0 2023-07-14T19:01:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e019fba0 con 0x7f84e01006b0 2023-07-14T19:01:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.958+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e01a0060 con 0x7f84e01006b0 2023-07-14T19:01:59.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.959+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e0106120 con 0x7f84e01006b0 2023-07-14T19:01:59.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.960+0000 7f84dcff9700 1 -- 172.21.15.78:0/3192901154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f84d800db20 con 0x7f84e01006b0 2023-07-14T19:01:59.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.960+0000 7f84dcff9700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f84c80a12a0 0x7f84c80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:01:59.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.961+0000 7f84df7fe700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f84c80a12a0 0x7f84c80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:01:59.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.961+0000 7f84dcff9700 1 -- 172.21.15.78:0/3192901154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f84d80cbd50 con 0x7f84e01006b0 2023-07-14T19:01:59.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.961+0000 7f84df7fe700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f84c80a12a0 0x7f84c80a3750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f84d000b2e0 tx=0x7f84d0004500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:01:59.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:01:59.965+0000 7f84dcff9700 1 -- 172.21.15.78:0/3192901154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f84d804e780 con 0x7f84e01006b0 2023-07-14T19:02:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.194+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84e002c8e0 con 0x7f84e01006b0 2023-07-14T19:02:00.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.194+0000 7f84dcff9700 1 -- 172.21.15.78:0/3192901154 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+501 (secure 0 0 0) 0x7f84d8095030 con 0x7f84e01006b0 2023-07-14T19:02:00.198 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:00.199 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2023-07-14T19:02:00.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f84c80a12a0 msgr2=0x7f84c80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:00.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f84c80a12a0 0x7f84c80a3750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f84d000b2e0 tx=0x7f84d0004500 comp rx=0 tx=0).stop 2023-07-14T19:02:00.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e01006b0 msgr2=0x7f84e019eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:00.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e01006b0 0x7f84e019eee0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f84d8007f10 tx=0x7f84d800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:00.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 shutdown_connections 2023-07-14T19:02:00.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f84c80a12a0 0x7f84c80a3750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:00.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f84e01006b0 0x7f84e019eee0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:00.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84e00feb30 0x7f84e0069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:00.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 --2- 172.21.15.78:0/3192901154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f84e00ffd20 0x7f84e019c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:00.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.196+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 >> 172.21.15.78:0/3192901154 conn(0x7f84e00fa150 msgr2=0x7f84e0103260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:00.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.197+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 shutdown_connections 2023-07-14T19:02:00.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:00.197+0000 7f84e5b2f700 1 -- 172.21.15.78:0/3192901154 wait complete. 2023-07-14T19:02:00.202 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 1 2023-07-14T19:02:00.381 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:02:00 smithi102 systemd[1]: Starting Ceph mds.l for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:00.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:00 smithi102 ceph-mon[129280]: Deploying daemon mds.l on smithi102 2023-07-14T19:02:00.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:00 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1011135600' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:00.732 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:00 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/27962644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:00.733 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:00 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3192901154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:00.733 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:02:00 smithi102 podman[154432]: 2023-07-14T19:02:00.733 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:02:00 smithi102 bash[154432]: eaa63461d941aa88f5ffc013dbc9aa889b562c57a2cdfe6f634ef9b67b7f848d 2023-07-14T19:02:00.733 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:02:00 smithi102 ceph-mds[154473]: -- 172.21.15.102:0/2403311254 <== mon.1 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55789f9cf1e0 con 0x55789f9a4800 2023-07-14T19:02:00.733 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:02:00 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-l[154448]: starting mds.l at 2023-07-14T19:02:00.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:00 smithi078 ceph-mon[127425]: Deploying daemon mds.l on smithi102 2023-07-14T19:02:00.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:00 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1011135600' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:00.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:00 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/27962644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:00.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:00 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3192901154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:00.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:00 smithi088 ceph-mon[128973]: Deploying daemon mds.l on smithi102 2023-07-14T19:02:00.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:00 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1011135600' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:00.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:00 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/27962644' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:00.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:00 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3192901154' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:01.131 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:02:00 smithi102 systemd[1]: Started Ceph mds.l for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:01.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:01.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.102:6832/2462741496,v1:172.21.15.102:6833/2462741496] up:boot 2023-07-14T19:02:01.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: fsmap 1 up:standby 2023-07-14T19:02:01.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:02:01.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:01.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:01.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:01 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:01.803 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:01.803 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.102:6832/2462741496,v1:172.21.15.102:6833/2462741496] up:boot 2023-07-14T19:02:01.803 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: fsmap 1 up:standby 2023-07-14T19:02:01.803 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:02:01.803 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:01.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:01.804 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:01 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:01.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:01.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.102:6832/2462741496,v1:172.21.15.102:6833/2462741496] up:boot 2023-07-14T19:02:01.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: fsmap 1 up:standby 2023-07-14T19:02:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2023-07-14T19:02:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:01.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:01 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:02.213 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:02.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 -- 172.21.15.78:0/3545233309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 msgr2=0x7f9d04106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:02.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 --2- 172.21.15.78:0/3545233309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 0x7f9d04106f40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00b0a0 tx=0x7f9cfc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 -- 172.21.15.78:0/3545233309 shutdown_connections 2023-07-14T19:02:02.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 --2- 172.21.15.78:0/3545233309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d040feb00 0x7f9d040fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 --2- 172.21.15.78:0/3545233309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 0x7f9d04106f40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 --2- 172.21.15.78:0/3545233309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d040ffcf0 0x7f9d04100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 -- 172.21.15.78:0/3545233309 >> 172.21.15.78:0/3545233309 conn(0x7f9d040fa100 msgr2=0x7f9d040fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:02.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 -- 172.21.15.78:0/3545233309 shutdown_connections 2023-07-14T19:02:02.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.346+0000 7f9d0918a700 1 -- 172.21.15.78:0/3545233309 wait complete. 2023-07-14T19:02:02.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.347+0000 7f9d0918a700 1 Processor -- start 2023-07-14T19:02:02.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.347+0000 7f9d0918a700 1 -- start start 2023-07-14T19:02:02.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0918a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d040feb00 0x7f9d041947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:02.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0918a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d040ffcf0 0x7f9d04194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:02.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0918a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 0x7f9d0419bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:02.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0918a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0419c370 con 0x7f9d04100680 2023-07-14T19:02:02.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0918a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0419c4b0 con 0x7f9d040ffcf0 2023-07-14T19:02:02.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0918a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d0419c640 con 0x7f9d040feb00 2023-07-14T19:02:02.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 0x7f9d0419bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:02.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 0x7f9d0419bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34170/0 (socket says 172.21.15.78:34170) 2023-07-14T19:02:02.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 -- 172.21.15.78:0/337356103 learned_addr learned my addr 172.21.15.78:0/337356103 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:02.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0259c700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d040ffcf0 0x7f9d04194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:02.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d02d9d700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d040feb00 0x7f9d041947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:02.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 -- 172.21.15.78:0/337356103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d040feb00 msgr2=0x7f9d041947f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:02.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d040feb00 0x7f9d041947f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 -- 172.21.15.78:0/337356103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d040ffcf0 msgr2=0x7f9d04194d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:02.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d040ffcf0 0x7f9d04194d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 -- 172.21.15.78:0/337356103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cfc009d00 con 0x7f9d04100680 2023-07-14T19:02:02.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0259c700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d040ffcf0 0x7f9d04194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:02:02.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0359e700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 0x7f9d0419bda0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00bad0 tx=0x7f9cfc00bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:02.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9cfbfff700 1 -- 172.21.15.78:0/337356103 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc00e040 con 0x7f9d04100680 2023-07-14T19:02:02.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d0419c8c0 con 0x7f9d04100680 2023-07-14T19:02:02.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9cfbfff700 1 -- 172.21.15.78:0/337356103 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9cfc003c00 con 0x7f9d04100680 2023-07-14T19:02:02.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9cfbfff700 1 -- 172.21.15.78:0/337356103 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cfc0052a0 con 0x7f9d04100680 2023-07-14T19:02:02.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.348+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d0419cd90 con 0x7f9d04100680 2023-07-14T19:02:02.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.349+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d04106070 con 0x7f9d04100680 2023-07-14T19:02:02.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.350+0000 7f9cfbfff700 1 -- 172.21.15.78:0/337356103 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9cfc01d470 con 0x7f9d04100680 2023-07-14T19:02:02.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.351+0000 7f9cfbfff700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9cf00a11a0 0x7f9cf00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:02.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.351+0000 7f9d02d9d700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9cf00a11a0 0x7f9cf00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:02.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.351+0000 7f9cfbfff700 1 -- 172.21.15.78:0/337356103 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f9cfc0cdee0 con 0x7f9d04100680 2023-07-14T19:02:02.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.351+0000 7f9d02d9d700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9cf00a11a0 0x7f9cf00a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9d040ffb50 tx=0x7f9cf4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:02.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.354+0000 7f9cfbfff700 1 -- 172.21.15.78:0/337356103 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9cfc097b60 con 0x7f9d04100680 2023-07-14T19:02:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:02 smithi078 ceph-mon[127425]: Deploying daemon mds.k on smithi088 2023-07-14T19:02:02.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.585+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d0402c8e0 con 0x7f9d04100680 2023-07-14T19:02:02.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.585+0000 7f9cfbfff700 1 -- 172.21.15.78:0/337356103 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1260 (secure 0 0 0) 0x7f9cfc09b180 con 0x7f9d04100680 2023-07-14T19:02:02.590 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:02.590 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2}],"filesystems":[]} 2023-07-14T19:02:02.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.587+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9cf00a11a0 msgr2=0x7f9cf00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:02.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.587+0000 7f9d0918a700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9cf00a11a0 0x7f9cf00a3650 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9d040ffb50 tx=0x7f9cf4009260 comp rx=0 tx=0).stop 2023-07-14T19:02:02.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.587+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 msgr2=0x7f9d0419bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:02.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.587+0000 7f9d0918a700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 0x7f9d0419bda0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9cfc00bad0 tx=0x7f9cfc00bb00 comp rx=0 tx=0).stop 2023-07-14T19:02:02.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.588+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 shutdown_connections 2023-07-14T19:02:02.593 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.588+0000 7f9d0918a700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9cf00a11a0 0x7f9cf00a3650 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.588+0000 7f9d0918a700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9d04100680 0x7f9d0419bda0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.588+0000 7f9d0918a700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d040feb00 0x7f9d041947f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.588+0000 7f9d0918a700 1 --2- 172.21.15.78:0/337356103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9d040ffcf0 0x7f9d04194d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:02.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.588+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 >> 172.21.15.78:0/337356103 conn(0x7f9d040fa100 msgr2=0x7f9d041031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:02.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.588+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 shutdown_connections 2023-07-14T19:02:02.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:02.588+0000 7f9d0918a700 1 -- 172.21.15.78:0/337356103 wait complete. 2023-07-14T19:02:02.595 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 2 2023-07-14T19:02:02.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:02 smithi088 ceph-mon[128973]: Deploying daemon mds.k on smithi088 2023-07-14T19:02:02.684 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:02:02 smithi088 systemd[1]: Starting Ceph mds.k for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:02.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:02 smithi102 ceph-mon[129280]: Deploying daemon mds.k on smithi088 2023-07-14T19:02:02.962 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:02:02 smithi088 podman[153085]: 2023-07-14T19:02:03.462 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:02:02 smithi088 ceph-mds[153125]: -- 172.21.15.88:0/1071527772 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56081a1b91e0 con 0x56081a18ec00 2023-07-14T19:02:03.462 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:02:02 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-k[153101]: starting mds.k at 2023-07-14T19:02:03.462 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:02:02 smithi088 bash[153085]: c700cf98ed5f5976e27c3b5dc80b7dc229e9c1c01300914b9559347753683397 2023-07-14T19:02:03.463 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:02:03 smithi088 systemd[1]: Started Ceph mds.k for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:03.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:03 smithi078 ceph-mon[127425]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:03.762 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:03 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/337356103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:03.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:03.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:03.763 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:03 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:03.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:03 smithi102 ceph-mon[129280]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:03.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:03 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/337356103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:03.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:03.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:03.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:03 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:03.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:03 smithi088 ceph-mon[128973]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:03.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:03 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/337356103' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:03.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:03.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:03.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:03.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:03 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:04.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:04.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 -- 172.21.15.78:0/1521549664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0feb10 msgr2=0x7f5ecc0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:04.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1521549664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0feb10 0x7f5ecc0fef20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc009990 tx=0x7f5ebc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:04.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 -- 172.21.15.78:0/1521549664 shutdown_connections 2023-07-14T19:02:04.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1521549664 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ecc100690 0x7f5ecc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:04.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1521549664 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0feb10 0x7f5ecc0fef20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:04.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1521549664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ecc0ffd00 0x7f5ecc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:04.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 -- 172.21.15.78:0/1521549664 >> 172.21.15.78:0/1521549664 conn(0x7f5ecc0fa130 msgr2=0x7f5ecc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:04.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 -- 172.21.15.78:0/1521549664 shutdown_connections 2023-07-14T19:02:04.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.749+0000 7f5ed1931700 1 -- 172.21.15.78:0/1521549664 wait complete. 2023-07-14T19:02:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.750+0000 7f5ed1931700 1 Processor -- start 2023-07-14T19:02:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.750+0000 7f5ed1931700 1 -- start start 2023-07-14T19:02:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.750+0000 7f5ed1931700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ecc0feb10 0x7f5ecc1947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.750+0000 7f5ed1931700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0ffd00 0x7f5ecc194d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.750+0000 7f5ed1931700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ecc100690 0x7f5ecc19bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.750+0000 7f5ed1931700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc19c370 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.750+0000 7f5ed1931700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc19c4b0 con 0x7f5ecc0feb10 2023-07-14T19:02:04.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.750+0000 7f5ed1931700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ecc19c640 con 0x7f5ecc100690 2023-07-14T19:02:04.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0ffd00 0x7f5ecc194d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:04.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0ffd00 0x7f5ecc194d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34176/0 (socket says 172.21.15.78:34176) 2023-07-14T19:02:04.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 -- 172.21.15.78:0/1124667589 learned_addr learned my addr 172.21.15.78:0/1124667589 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:04.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5ecaffd700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ecc0feb10 0x7f5ecc1947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:04.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5ecb7fe700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ecc100690 0x7f5ecc19bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:04.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 -- 172.21.15.78:0/1124667589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ecc100690 msgr2=0x7f5ecc19bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:04.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ecc100690 0x7f5ecc19bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:04.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 -- 172.21.15.78:0/1124667589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ecc0feb10 msgr2=0x7f5ecc1947f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:04.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ecc0feb10 0x7f5ecc1947f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:04.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 -- 172.21.15.78:0/1124667589 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ebc0095f0 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5ecb7fe700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ecc100690 0x7f5ecc19bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:04.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5eca7fc700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0ffd00 0x7f5ecc194d30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000e9a0 tx=0x7f5ec000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:04.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5ed092f700 1 -- 172.21.15.78:0/1124667589 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0009790 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5ed092f700 1 -- 172.21.15.78:0/1124667589 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5ec000cbd0 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ecc19c920 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5ed092f700 1 -- 172.21.15.78:0/1124667589 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec0018680 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.751+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ecc19ce30 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.752+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ecc10a1e0 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.753+0000 7f5ed092f700 1 -- 172.21.15.78:0/1124667589 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5ec000cd40 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.754+0000 7f5ed092f700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb40a11a0 0x7f5eb40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:04.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.754+0000 7f5ecaffd700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb40a11a0 0x7f5eb40a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:04.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.754+0000 7f5ed092f700 1 -- 172.21.15.78:0/1124667589 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5ec0014070 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.754+0000 7f5ecaffd700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb40a11a0 0x7f5eb40a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0098e0 tx=0x7f5ebc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:04.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.757+0000 7f5ed092f700 1 -- 172.21.15.78:0/1124667589 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5ec00966c0 con 0x7f5ecc0ffd00 2023-07-14T19:02:04.797 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:02:04 smithi078 systemd[1]: Starting Ceph mds.j for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:04 smithi078 ceph-mon[127425]: Deploying daemon mds.j on smithi078 2023-07-14T19:02:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:04 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6832/244273805,v1:172.21.15.88:6833/244273805] up:boot 2023-07-14T19:02:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:04 smithi078 ceph-mon[127425]: fsmap 2 up:standby 2023-07-14T19:02:04.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:04 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:02:04.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:04 smithi102 ceph-mon[129280]: Deploying daemon mds.j on smithi078 2023-07-14T19:02:04.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:04 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6832/244273805,v1:172.21.15.88:6833/244273805] up:boot 2023-07-14T19:02:04.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:04 smithi102 ceph-mon[129280]: fsmap 2 up:standby 2023-07-14T19:02:04.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:04 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:02:04.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:04 smithi088 ceph-mon[128973]: Deploying daemon mds.j on smithi078 2023-07-14T19:02:04.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:04 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6832/244273805,v1:172.21.15.88:6833/244273805] up:boot 2023-07-14T19:02:04.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:04 smithi088 ceph-mon[128973]: fsmap 2 up:standby 2023-07-14T19:02:04.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:04 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2023-07-14T19:02:05.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.998+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ecc105ea0 con 0x7f5ecc0ffd00 2023-07-14T19:02:05.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:04.999+0000 7f5ed092f700 1 -- 172.21.15.78:0/1124667589 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2014 (secure 0 0 0) 0x7f5ec0099ce0 con 0x7f5ecc0ffd00 2023-07-14T19:02:05.003 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:05.003 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-07-14T19:02:05.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb40a11a0 msgr2=0x7f5eb40a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:05.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb40a11a0 0x7f5eb40a3650 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc0098e0 tx=0x7f5ebc00b260 comp rx=0 tx=0).stop 2023-07-14T19:02:05.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0ffd00 msgr2=0x7f5ecc194d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:05.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0ffd00 0x7f5ecc194d30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5ec000e9a0 tx=0x7f5ec000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:02:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 shutdown_connections 2023-07-14T19:02:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb40a11a0 0x7f5eb40a3650 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ecc0feb10 0x7f5ecc1947f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ecc100690 0x7f5ecc19bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 --2- 172.21.15.78:0/1124667589 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ecc0ffd00 0x7f5ecc194d30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 >> 172.21.15.78:0/1124667589 conn(0x7f5ecc0fa130 msgr2=0x7f5ecc1031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:05.006 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 shutdown_connections 2023-07-14T19:02:05.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:05.001+0000 7f5ed1931700 1 -- 172.21.15.78:0/1124667589 wait complete. 2023-07-14T19:02:05.007 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 3 2023-07-14T19:02:05.290 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:02:04 smithi078 podman[160798]: 2023-07-14T19:02:05.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mds[160840]: -- 172.21.15.78:0/706377527 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e33b6611e0 con 0x55e33b636800 2023-07-14T19:02:05.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-j[160816]: starting mds.j at 2023-07-14T19:02:05.548 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:02:05 smithi078 bash[160798]: 29eaebd80c3d1b862e94ce32c1359087139d15c82a88201d0589d6734fc24ac1 2023-07-14T19:02:05.549 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:02:05 smithi078 systemd[1]: Started Ceph mds.j for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:05.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mon[127425]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:05.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1124667589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:05.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:05.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:05 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:05.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:05 smithi102 ceph-mon[129280]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:05.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:05 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1124667589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:05.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:05.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:05.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:05 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:05.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:05 smithi088 ceph-mon[128973]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:05.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:05 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1124667589' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:05.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:05.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:05.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:05.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:05 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:06.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:06 smithi078 ceph-mon[127425]: Deploying daemon mds.d on smithi078 2023-07-14T19:02:06.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:06 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.78:6834/2843510081,v1:172.21.15.78:6835/2843510081] up:boot 2023-07-14T19:02:06.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:06 smithi078 ceph-mon[127425]: fsmap 3 up:standby 2023-07-14T19:02:06.638 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:06 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:02:06.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:06 smithi102 ceph-mon[129280]: Deploying daemon mds.d on smithi078 2023-07-14T19:02:06.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:06 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.78:6834/2843510081,v1:172.21.15.78:6835/2843510081] up:boot 2023-07-14T19:02:06.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:06 smithi102 ceph-mon[129280]: fsmap 3 up:standby 2023-07-14T19:02:06.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:06 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:02:06.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:06 smithi088 ceph-mon[128973]: Deploying daemon mds.d on smithi078 2023-07-14T19:02:06.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:06 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.78:6834/2843510081,v1:172.21.15.78:6835/2843510081] up:boot 2023-07-14T19:02:06.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:06 smithi088 ceph-mon[128973]: fsmap 3 up:standby 2023-07-14T19:02:06.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:06 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2023-07-14T19:02:07.018 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:07.047 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:02:06 smithi078 systemd[1]: Starting Ceph mds.d for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:07.149 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.145+0000 7f822a06e700 1 -- 172.21.15.78:0/1944979553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 msgr2=0x7f82241004a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:07.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.145+0000 7f822a06e700 1 --2- 172.21.15.78:0/1944979553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 0x7f82241004a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8214009990 tx=0x7f8214009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.145+0000 7f822a06e700 1 -- 172.21.15.78:0/1944979553 shutdown_connections 2023-07-14T19:02:07.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.145+0000 7f822a06e700 1 --2- 172.21.15.78:0/1944979553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82240ff750 0x7f8224107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.145+0000 7f822a06e700 1 --2- 172.21.15.78:0/1944979553 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 0x7f82241004a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.150 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.145+0000 7f822a06e700 1 --2- 172.21.15.78:0/1944979553 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82240fedc0 0x7f82240ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.145+0000 7f822a06e700 1 -- 172.21.15.78:0/1944979553 >> 172.21.15.78:0/1944979553 conn(0x7f82240f98a0 msgr2=0x7f82240fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:07.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 -- 172.21.15.78:0/1944979553 shutdown_connections 2023-07-14T19:02:07.151 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 -- 172.21.15.78:0/1944979553 wait complete. 2023-07-14T19:02:07.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 Processor -- start 2023-07-14T19:02:07.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 -- start start 2023-07-14T19:02:07.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82240fedc0 0x7f8224194780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:07.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82240ff750 0x7f8224194cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:07.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 0x7f822419bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:07.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822419c300 con 0x7f8224100090 2023-07-14T19:02:07.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822419c440 con 0x7f82240ff750 2023-07-14T19:02:07.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.146+0000 7f822a06e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822419c5d0 con 0x7f82240fedc0 2023-07-14T19:02:07.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 0x7f822419bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:07.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 0x7f822419bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34208/0 (socket says 172.21.15.78:34208) 2023-07-14T19:02:07.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 -- 172.21.15.78:0/2762942733 learned_addr learned my addr 172.21.15.78:0/2762942733 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:07.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f82237fe700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82240fedc0 0x7f8224194780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:07.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f821affd700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82240ff750 0x7f8224194cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:07.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 -- 172.21.15.78:0/2762942733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82240fedc0 msgr2=0x7f8224194780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:07.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82240fedc0 0x7f8224194780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 -- 172.21.15.78:0/2762942733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82240ff750 msgr2=0x7f8224194cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:07.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82240ff750 0x7f8224194cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 -- 172.21.15.78:0/2762942733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82140095f0 con 0x7f8224100090 2023-07-14T19:02:07.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f8223fff700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 0x7f822419bd30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f821c007ee0 tx=0x7f821c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:07.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f82217fa700 1 -- 172.21.15.78:0/2762942733 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c00dd20 con 0x7f8224100090 2023-07-14T19:02:07.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f82217fa700 1 -- 172.21.15.78:0/2762942733 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f821c00f040 con 0x7f8224100090 2023-07-14T19:02:07.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822419c8b0 con 0x7f8224100090 2023-07-14T19:02:07.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f82217fa700 1 -- 172.21.15.78:0/2762942733 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f821c004750 con 0x7f8224100090 2023-07-14T19:02:07.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.147+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822419cdc0 con 0x7f8224100090 2023-07-14T19:02:07.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.148+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8224106220 con 0x7f8224100090 2023-07-14T19:02:07.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.149+0000 7f82217fa700 1 -- 172.21.15.78:0/2762942733 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f821c003a00 con 0x7f8224100090 2023-07-14T19:02:07.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.150+0000 7f82217fa700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82040a11a0 0x7f82040a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:07.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.150+0000 7f82217fa700 1 -- 172.21.15.78:0/2762942733 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f821c0cc550 con 0x7f8224100090 2023-07-14T19:02:07.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.150+0000 7f82237fe700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82040a11a0 0x7f82040a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:07.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.150+0000 7f82237fe700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82040a11a0 0x7f82040a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8214000c00 tx=0x7f821400b560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:07.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.153+0000 7f82217fa700 1 -- 172.21.15.78:0/2762942733 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f821c0960a0 con 0x7f8224100090 2023-07-14T19:02:07.315 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:02:07 smithi078 podman[161143]: 2023-07-14T19:02:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.395+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f822402c8e0 con 0x7f8224100090 2023-07-14T19:02:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.395+0000 7f82217fa700 1 -- 172.21.15.78:0/2762942733 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2771 (secure 0 0 0) 0x7f821c0048b0 con 0x7f8224100090 2023-07-14T19:02:07.400 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:07.400 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[]} 2023-07-14T19:02:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82040a11a0 msgr2=0x7f82040a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82040a11a0 0x7f82040a3650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8214000c00 tx=0x7f821400b560 comp rx=0 tx=0).stop 2023-07-14T19:02:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 msgr2=0x7f822419bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 0x7f822419bd30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f821c007ee0 tx=0x7f821c00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 shutdown_connections 2023-07-14T19:02:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f82040a11a0 0x7f82040a3650 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8224100090 0x7f822419bd30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f82240fedc0 0x7f8224194780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 --2- 172.21.15.78:0/2762942733 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f82240ff750 0x7f8224194cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.397+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 >> 172.21.15.78:0/2762942733 conn(0x7f82240f98a0 msgr2=0x7f8224103360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.398+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 shutdown_connections 2023-07-14T19:02:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:07.398+0000 7f822a06e700 1 -- 172.21.15.78:0/2762942733 wait complete. 2023-07-14T19:02:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 4 2023-07-14T19:02:07.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:07 smithi078 ceph-mon[127425]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:07.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:07 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2762942733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:07.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:07 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:07.798 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:02:07 smithi078 ceph-mds[161228]: -- 172.21.15.78:0/208607483 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55831b12d1e0 con 0x55831b103000 2023-07-14T19:02:07.798 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:02:07 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-d[161202]: starting mds.d at 2023-07-14T19:02:07.798 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:02:07 smithi078 bash[161143]: aa7b4c49d79171903881c0af98ffe804a6afe50534cad01d4d9b22e98acf2c65 2023-07-14T19:02:07.799 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:02:07 smithi078 systemd[1]: Started Ceph mds.d for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:07.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:07 smithi102 ceph-mon[129280]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:07.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:07 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2762942733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:07.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:07 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:07 smithi088 ceph-mon[128973]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:07 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2762942733' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:07.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:07 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:08.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: Deploying daemon mds.e on smithi088 2023-07-14T19:02:08.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.78:6836/4055842453,v1:172.21.15.78:6837/4055842453] up:boot 2023-07-14T19:02:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: fsmap 4 up:standby 2023-07-14T19:02:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:02:08.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:08 smithi078 ceph-mon[127425]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:08.800 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:08.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:08.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:08.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:08.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:08.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: Deploying daemon mds.e on smithi088 2023-07-14T19:02:08.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.78:6836/4055842453,v1:172.21.15.78:6837/4055842453] up:boot 2023-07-14T19:02:08.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: fsmap 4 up:standby 2023-07-14T19:02:08.801 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:02:08.802 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:08 smithi088 ceph-mon[128973]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:08.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:08.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:08.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:08.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:08.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:08.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: Deploying daemon mds.e on smithi088 2023-07-14T19:02:08.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.78:6836/4055842453,v1:172.21.15.78:6837/4055842453] up:boot 2023-07-14T19:02:08.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: fsmap 4 up:standby 2023-07-14T19:02:08.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2023-07-14T19:02:08.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:08 smithi102 ceph-mon[129280]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:09.129 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:02:09 smithi088 systemd[1]: Starting Ceph mds.e for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:09.415 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:09.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 -- 172.21.15.78:0/4035269168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 msgr2=0x7efd840fef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:09.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 --2- 172.21.15.78:0/4035269168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 0x7efd840fef30 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7efd74009990 tx=0x7efd74009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 -- 172.21.15.78:0/4035269168 shutdown_connections 2023-07-14T19:02:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 --2- 172.21.15.78:0/4035269168 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd841006a0 0x7efd84106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 --2- 172.21.15.78:0/4035269168 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 0x7efd840fef30 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 --2- 172.21.15.78:0/4035269168 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840ffd10 0x7efd84100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 -- 172.21.15.78:0/4035269168 >> 172.21.15.78:0/4035269168 conn(0x7efd840fa100 msgr2=0x7efd840fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 -- 172.21.15.78:0/4035269168 shutdown_connections 2023-07-14T19:02:09.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 -- 172.21.15.78:0/4035269168 wait complete. 2023-07-14T19:02:09.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.548+0000 7efd8c18d700 1 Processor -- start 2023-07-14T19:02:09.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8c18d700 1 -- start start 2023-07-14T19:02:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8c18d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 0x7efd84069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8c18d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840ffd10 0x7efd8419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:09.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8c18d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd841006a0 0x7efd8419cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:09.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8c18d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8419d360 con 0x7efd840feb20 2023-07-14T19:02:09.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8c18d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8419d4d0 con 0x7efd840ffd10 2023-07-14T19:02:09.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8c18d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8419d690 con 0x7efd841006a0 2023-07-14T19:02:09.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 0x7efd84069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:09.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 0x7efd84069360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34232/0 (socket says 172.21.15.78:34232) 2023-07-14T19:02:09.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 -- 172.21.15.78:0/1210123576 learned_addr learned my addr 172.21.15.78:0/1210123576 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:09.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89728700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840ffd10 0x7efd8419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:09.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8a72a700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd841006a0 0x7efd8419cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:09.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 -- 172.21.15.78:0/1210123576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd841006a0 msgr2=0x7efd8419cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:09.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd841006a0 0x7efd8419cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 -- 172.21.15.78:0/1210123576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840ffd10 msgr2=0x7efd8419a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840ffd10 0x7efd8419a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 -- 172.21.15.78:0/1210123576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd740095f0 con 0x7efd840feb20 2023-07-14T19:02:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8a72a700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd841006a0 0x7efd8419cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd89f29700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 0x7efd84069360 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efd74000c00 tx=0x7efd74009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd7affd700 1 -- 172.21.15.78:0/1210123576 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd7401d070 con 0x7efd840feb20 2023-07-14T19:02:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd7affd700 1 -- 172.21.15.78:0/1210123576 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efd740040b0 con 0x7efd840feb20 2023-07-14T19:02:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.549+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd8419d910 con 0x7efd840feb20 2023-07-14T19:02:09.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.550+0000 7efd7affd700 1 -- 172.21.15.78:0/1210123576 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd74022cb0 con 0x7efd840feb20 2023-07-14T19:02:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.550+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd8419ddc0 con 0x7efd840feb20 2023-07-14T19:02:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.550+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd8410a1a0 con 0x7efd840feb20 2023-07-14T19:02:09.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.551+0000 7efd7affd700 1 -- 172.21.15.78:0/1210123576 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7efd74022470 con 0x7efd840feb20 2023-07-14T19:02:09.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.552+0000 7efd7affd700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efd700a11a0 0x7efd700a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:09.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.552+0000 7efd89728700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efd700a11a0 0x7efd700a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:09.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.552+0000 7efd7affd700 1 -- 172.21.15.78:0/1210123576 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7efd740ce040 con 0x7efd840feb20 2023-07-14T19:02:09.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.553+0000 7efd89728700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efd700a11a0 0x7efd700a3650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efd80005950 tx=0x7efd800058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:09.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.556+0000 7efd7affd700 1 -- 172.21.15.78:0/1210123576 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efd74097cc0 con 0x7efd840feb20 2023-07-14T19:02:09.700 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:02:09 smithi088 podman[153427]: 2023-07-14T19:02:09.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.792+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd84105a60 con 0x7efd840feb20 2023-07-14T19:02:09.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.793+0000 7efd7affd700 1 -- 172.21.15.78:0/1210123576 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4285 (secure 0 0 0) 0x7efd7409b2e0 con 0x7efd840feb20 2023-07-14T19:02:09.798 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:09.798 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T19:02:09.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efd700a11a0 msgr2=0x7efd700a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:09.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efd700a11a0 0x7efd700a3650 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7efd80005950 tx=0x7efd800058e0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 msgr2=0x7efd84069360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:09.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 0x7efd84069360 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7efd74000c00 tx=0x7efd74009f90 comp rx=0 tx=0).stop 2023-07-14T19:02:09.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 shutdown_connections 2023-07-14T19:02:09.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efd700a11a0 0x7efd700a3650 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efd841006a0 0x7efd8419cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efd840feb20 0x7efd84069360 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 --2- 172.21.15.78:0/1210123576 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efd840ffd10 0x7efd8419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:09.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 >> 172.21.15.78:0/1210123576 conn(0x7efd840fa100 msgr2=0x7efd840fc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:09.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 shutdown_connections 2023-07-14T19:02:09.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:09.795+0000 7efd8c18d700 1 -- 172.21.15.78:0/1210123576 wait complete. 2023-07-14T19:02:09.801 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 6 2023-07-14T19:02:09.962 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:02:09 smithi088 bash[153427]: 7ea1cb288dd64628471f9ebc7d5a293f4e6561ac342cb0c7cfb3567e5e9334ca 2023-07-14T19:02:09.962 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:02:09 smithi088 ceph-mds[153469]: -- 172.21.15.88:0/3896102434 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558c28f691e0 con 0x558c28f3ec00 2023-07-14T19:02:09.962 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:02:09 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-e[153444]: starting mds.e at 2023-07-14T19:02:09.962 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:02:09 smithi088 systemd[1]: Started Ceph mds.e for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6834/3667124043,v1:172.21.15.88:6835/3667124043] up:boot 2023-07-14T19:02:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: fsmap 5 up:standby 2023-07-14T19:02:11.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:02:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1210123576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:11.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:11.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: Deploying daemon mds.f on smithi102 2023-07-14T19:02:11.050 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:10 smithi078 ceph-mon[127425]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:11.100 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6834/3667124043,v1:172.21.15.88:6835/3667124043] up:boot 2023-07-14T19:02:11.101 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: fsmap 5 up:standby 2023-07-14T19:02:11.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:02:11.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1210123576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:11.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.102 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:11.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:11.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:11.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: Deploying daemon mds.f on smithi102 2023-07-14T19:02:11.103 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:10 smithi102 ceph-mon[129280]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:11.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6834/3667124043,v1:172.21.15.88:6835/3667124043] up:boot 2023-07-14T19:02:11.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: fsmap 5 up:standby 2023-07-14T19:02:11.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:02:11.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1210123576' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:11.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:11.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:11.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:11.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:11.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: Deploying daemon mds.f on smithi102 2023-07-14T19:02:11.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:10 smithi088 ceph-mon[128973]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:11.381 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:02:11 smithi102 systemd[1]: Starting Ceph mds.f for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:11.811 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:11.925 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:02:11 smithi102 podman[154772]: 2023-07-14T19:02:11.925 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:02:11 smithi102 bash[154772]: 383a23e43256ef383f1a7f814f224eab261a72016048c8d553c5c60b496e8bbd 2023-07-14T19:02:11.925 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:02:11 smithi102 ceph-mds[154812]: -- 172.21.15.102:0/1991308950 <== mon.1 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e4ccdb1e0 con 0x564e4ccb0c00 2023-07-14T19:02:11.926 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:02:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-f[154788]: starting mds.f at 2023-07-14T19:02:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 -- 172.21.15.78:0/2522788152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc1000b0 msgr2=0x7f12fc1004c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 --2- 172.21.15.78:0/2522788152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc1000b0 0x7f12fc1004c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f12f8009990 tx=0x7f12f8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 -- 172.21.15.78:0/2522788152 shutdown_connections 2023-07-14T19:02:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 --2- 172.21.15.78:0/2522788152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12fc0ff6b0 0x7f12fc106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:11.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 --2- 172.21.15.78:0/2522788152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc1000b0 0x7f12fc1004c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 --2- 172.21.15.78:0/2522788152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12fc0fed00 0x7f12fc0ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 -- 172.21.15.78:0/2522788152 >> 172.21.15.78:0/2522788152 conn(0x7f12fc0f98a0 msgr2=0x7f12fc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 -- 172.21.15.78:0/2522788152 shutdown_connections 2023-07-14T19:02:11.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 -- 172.21.15.78:0/2522788152 wait complete. 2023-07-14T19:02:11.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 Processor -- start 2023-07-14T19:02:11.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 -- start start 2023-07-14T19:02:11.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fed00 0x7f12fc194710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:11.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12fc0ff6b0 0x7f12fc194c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:11.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12fc1000b0 0x7f12fc19bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:11.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc19c290 con 0x7f12fc0fed00 2023-07-14T19:02:11.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.946+0000 7f130467d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc19c3d0 con 0x7f12fc0ff6b0 2023-07-14T19:02:11.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f130467d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12fc19c560 con 0x7f12fc1000b0 2023-07-14T19:02:11.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fed00 0x7f12fc194710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:11.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fed00 0x7f12fc194710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34246/0 (socket says 172.21.15.78:34246) 2023-07-14T19:02:11.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 -- 172.21.15.78:0/3958452215 learned_addr learned my addr 172.21.15.78:0/3958452215 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:11.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302c1a700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12fc1000b0 0x7f12fc19bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:11.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1301c18700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12fc0ff6b0 0x7f12fc194c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 -- 172.21.15.78:0/3958452215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12fc1000b0 msgr2=0x7f12fc19bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12fc1000b0 0x7f12fc19bcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 -- 172.21.15.78:0/3958452215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12fc0ff6b0 msgr2=0x7f12fc194c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12fc0ff6b0 0x7f12fc194c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 -- 172.21.15.78:0/3958452215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f80095f0 con 0x7f12fc0fed00 2023-07-14T19:02:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1301c18700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12fc0ff6b0 0x7f12fc194c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:11.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f1302419700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fed00 0x7f12fc194710 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f12f80098e0 tx=0x7f12f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f12f37fe700 1 -- 172.21.15.78:0/3958452215 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f801d070 con 0x7f12fc0fed00 2023-07-14T19:02:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f12f37fe700 1 -- 172.21.15.78:0/3958452215 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f12f800be30 con 0x7f12fc0fed00 2023-07-14T19:02:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12fc19c7e0 con 0x7f12fc0fed00 2023-07-14T19:02:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f12f37fe700 1 -- 172.21.15.78:0/3958452215 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f8022e00 con 0x7f12fc0fed00 2023-07-14T19:02:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.947+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12fc19cc60 con 0x7f12fc0fed00 2023-07-14T19:02:11.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.948+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12fc10a1b0 con 0x7f12fc0fed00 2023-07-14T19:02:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.950+0000 7f12f37fe700 1 -- 172.21.15.78:0/3958452215 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f12f800b9a0 con 0x7f12fc0fed00 2023-07-14T19:02:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.950+0000 7f12f37fe700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12e80a1220 0x7f12e80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.950+0000 7f1301c18700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12e80a1220 0x7f12e80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:11.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.951+0000 7f12f37fe700 1 -- 172.21.15.78:0/3958452215 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12f80cdc90 con 0x7f12fc0fed00 2023-07-14T19:02:11.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.951+0000 7f1301c18700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12e80a1220 0x7f12e80a36d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f12ec005fd0 tx=0x7f12ec005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:11.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:11.956+0000 7f12f37fe700 1 -- 172.21.15.78:0/3958452215 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f12f8097910 con 0x7f12fc0fed00 2023-07-14T19:02:12.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.192+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12fc105a40 con 0x7f12fc0fed00 2023-07-14T19:02:12.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.192+0000 7f12f37fe700 1 -- 172.21.15.78:0/3958452215 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5039 (secure 0 0 0) 0x7f12f809af30 con 0x7f12fc0fed00 2023-07-14T19:02:12.197 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:12.197 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6}],"filesystems":[]} 2023-07-14T19:02:12.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12e80a1220 msgr2=0x7f12e80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:12.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12e80a1220 0x7f12e80a36d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f12ec005fd0 tx=0x7f12ec005e20 comp rx=0 tx=0).stop 2023-07-14T19:02:12.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fed00 msgr2=0x7f12fc194710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:12.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fed00 0x7f12fc194710 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f12f80098e0 tx=0x7f12f8004730 comp rx=0 tx=0).stop 2023-07-14T19:02:12.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 shutdown_connections 2023-07-14T19:02:12.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12e80a1220 0x7f12e80a36d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:12.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12fc1000b0 0x7f12fc19bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:12.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f12fc0fed00 0x7f12fc194710 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:12.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 --2- 172.21.15.78:0/3958452215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f12fc0ff6b0 0x7f12fc194c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:12.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 >> 172.21.15.78:0/3958452215 conn(0x7f12fc0f98a0 msgr2=0x7f12fc0fbaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:12.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 shutdown_connections 2023-07-14T19:02:12.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:12.195+0000 7f130467d700 1 -- 172.21.15.78:0/3958452215 wait complete. 2023-07-14T19:02:12.201 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 7 2023-07-14T19:02:12.381 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:02:11 smithi102 systemd[1]: Started Ceph mds.f for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:13.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.102:6834/75062394,v1:172.21.15.102:6835/75062394] up:boot 2023-07-14T19:02:13.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: fsmap 6 up:standby 2023-07-14T19:02:13.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:02:13.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:13.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:13.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:13.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: Deploying daemon mds.c on smithi102 2023-07-14T19:02:13.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:13.133 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:12 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3958452215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:13.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.102:6834/75062394,v1:172.21.15.102:6835/75062394] up:boot 2023-07-14T19:02:13.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: fsmap 6 up:standby 2023-07-14T19:02:13.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:02:13.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:13.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:13.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:13.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: Deploying daemon mds.c on smithi102 2023-07-14T19:02:13.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:13.214 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:12 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3958452215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:13.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.102:6834/75062394,v1:172.21.15.102:6835/75062394] up:boot 2023-07-14T19:02:13.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: fsmap 6 up:standby 2023-07-14T19:02:13.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:02:13.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:13.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:13.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:13.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:13.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: Deploying daemon mds.c on smithi102 2023-07-14T19:02:13.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:13.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:12 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3958452215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:13.631 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:02:13 smithi102 systemd[1]: Starting Ceph mds.c for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:14.017 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:02:13 smithi102 podman[155113]: 2023-07-14T19:02:14.018 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:02:13 smithi102 ceph-mds[155153]: -- 172.21.15.102:0/4107752395 <== mon.1 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f5fc6c51e0 con 0x55f5fc69a800 2023-07-14T19:02:14.018 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:02:13 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-c[155129]: starting mds.c at 2023-07-14T19:02:14.211 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:14.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 -- 172.21.15.78:0/4084197139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 msgr2=0x7fe5dc0ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:14.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/4084197139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 0x7fe5dc0ff1b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe5c8009990 tx=0x7fe5c8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 -- 172.21.15.78:0/4084197139 shutdown_connections 2023-07-14T19:02:14.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/4084197139 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5dc100180 0x7fe5dc100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/4084197139 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5dc0ff6f0 0x7fe5dc106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/4084197139 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 0x7fe5dc0ff1b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 -- 172.21.15.78:0/4084197139 >> 172.21.15.78:0/4084197139 conn(0x7fe5dc0f98a0 msgr2=0x7fe5dc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 -- 172.21.15.78:0/4084197139 shutdown_connections 2023-07-14T19:02:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.346+0000 7fe5e30e4700 1 -- 172.21.15.78:0/4084197139 wait complete. 2023-07-14T19:02:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e30e4700 1 Processor -- start 2023-07-14T19:02:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e30e4700 1 -- start start 2023-07-14T19:02:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e30e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 0x7fe5dc195800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e30e4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5dc0ff6f0 0x7fe5dc195d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:14.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e30e4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5dc100180 0x7fe5dc19cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e30e4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc19d380 con 0x7fe5dc0fed40 2023-07-14T19:02:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e30e4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc19d4c0 con 0x7fe5dc100180 2023-07-14T19:02:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e30e4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5dc19d650 con 0x7fe5dc0ff6f0 2023-07-14T19:02:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e0e80700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 0x7fe5dc195800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e0e80700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 0x7fe5dc195800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34260/0 (socket says 172.21.15.78:34260) 2023-07-14T19:02:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e0e80700 1 -- 172.21.15.78:0/1788619067 learned_addr learned my addr 172.21.15.78:0/1788619067 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:14.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e1681700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5dc100180 0x7fe5dc19cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:14.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5dbfff700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5dc0ff6f0 0x7fe5dc195d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:14.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e0e80700 1 -- 172.21.15.78:0/1788619067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5dc0ff6f0 msgr2=0x7fe5dc195d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:14.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e0e80700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5dc0ff6f0 0x7fe5dc195d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e0e80700 1 -- 172.21.15.78:0/1788619067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5dc100180 msgr2=0x7fe5dc19cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e0e80700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5dc100180 0x7fe5dc19cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5e0e80700 1 -- 172.21.15.78:0/1788619067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c80095f0 con 0x7fe5dc0fed40 2023-07-14T19:02:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.347+0000 7fe5dbfff700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5dc0ff6f0 0x7fe5dc195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.348+0000 7fe5e0e80700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 0x7fe5dc195800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000e9a0 tx=0x7fe5d000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.348+0000 7fe5d9ffb700 1 -- 172.21.15.78:0/1788619067 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0009790 con 0x7fe5dc0fed40 2023-07-14T19:02:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.348+0000 7fe5d9ffb700 1 -- 172.21.15.78:0/1788619067 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe5d000cbd0 con 0x7fe5dc0fed40 2023-07-14T19:02:14.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.348+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5dc19d930 con 0x7fe5dc0fed40 2023-07-14T19:02:14.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.348+0000 7fe5d9ffb700 1 -- 172.21.15.78:0/1788619067 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5d0018680 con 0x7fe5dc0fed40 2023-07-14T19:02:14.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.348+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5dc19de40 con 0x7fe5dc0fed40 2023-07-14T19:02:14.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.349+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5dc10a180 con 0x7fe5dc0fed40 2023-07-14T19:02:14.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.350+0000 7fe5d9ffb700 1 -- 172.21.15.78:0/1788619067 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe5d000cd40 con 0x7fe5dc0fed40 2023-07-14T19:02:14.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.350+0000 7fe5d9ffb700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5cc0a11a0 0x7fe5cc0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:14.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.351+0000 7fe5d9ffb700 1 -- 172.21.15.78:0/1788619067 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe5d0014070 con 0x7fe5dc0fed40 2023-07-14T19:02:14.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.351+0000 7fe5dbfff700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5cc0a11a0 0x7fe5cc0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:14.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.351+0000 7fe5dbfff700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5cc0a11a0 0x7fe5cc0a3650 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80059d0 tx=0x7fe5c8005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:14.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.354+0000 7fe5d9ffb700 1 -- 172.21.15.78:0/1788619067 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe5d00966c0 con 0x7fe5dc0fed40 2023-07-14T19:02:14.381 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:02:14 smithi102 bash[155113]: aecba148408be905805e5401e54334450a082ed69a16c943f37feacb33bd3b50 2023-07-14T19:02:14.381 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:02:14 smithi102 systemd[1]: Started Ceph mds.c for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:14.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.583+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5dc10a340 con 0x7fe5dc0fed40 2023-07-14T19:02:14.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.583+0000 7fe5d9ffb700 1 -- 172.21.15.78:0/1788619067 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+5799 (secure 0 0 0) 0x7fe5d0099ce0 con 0x7fe5dc0fed40 2023-07-14T19:02:14.588 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:14.588 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8}],"filesystems":[]} 2023-07-14T19:02:14.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.585+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5cc0a11a0 msgr2=0x7fe5cc0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:14.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.585+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5cc0a11a0 0x7fe5cc0a3650 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe5c80059d0 tx=0x7fe5c8005940 comp rx=0 tx=0).stop 2023-07-14T19:02:14.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.585+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 msgr2=0x7fe5dc195800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:14.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.585+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 0x7fe5dc195800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fe5d000e9a0 tx=0x7fe5d000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.586+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 shutdown_connections 2023-07-14T19:02:14.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.586+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe5cc0a11a0 0x7fe5cc0a3650 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.586+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe5dc100180 0x7fe5dc19cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.586+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe5dc0fed40 0x7fe5dc195800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.586+0000 7fe5e30e4700 1 --2- 172.21.15.78:0/1788619067 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe5dc0ff6f0 0x7fe5dc195d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:14.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.586+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 >> 172.21.15.78:0/1788619067 conn(0x7fe5dc0f98a0 msgr2=0x7fe5dc0fdc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:14.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.586+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 shutdown_connections 2023-07-14T19:02:14.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:14.586+0000 7fe5e30e4700 1 -- 172.21.15.78:0/1788619067 wait complete. 2023-07-14T19:02:14.592 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 8 2023-07-14T19:02:15.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.102:6836/3933565644,v1:172.21.15.102:6837/3933565644] up:boot 2023-07-14T19:02:15.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: fsmap 7 up:standby 2023-07-14T19:02:15.125 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:02:15.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:15.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:15.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:15.126 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:15.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: Deploying daemon mds.h on smithi088 2023-07-14T19:02:15.127 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:14 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1788619067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.102:6836/3933565644,v1:172.21.15.102:6837/3933565644] up:boot 2023-07-14T19:02:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: fsmap 7 up:standby 2023-07-14T19:02:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:02:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:15.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:15.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:15.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:15.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: Deploying daemon mds.h on smithi088 2023-07-14T19:02:15.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:14 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1788619067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:15.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.102:6836/3933565644,v1:172.21.15.102:6837/3933565644] up:boot 2023-07-14T19:02:15.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: fsmap 7 up:standby 2023-07-14T19:02:15.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:02:15.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.382 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:15.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:15.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:15.383 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:15.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:15.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: Deploying daemon mds.h on smithi088 2023-07-14T19:02:15.384 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:14 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1788619067' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:15.861 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:02:15 smithi088 systemd[1]: Starting Ceph mds.h for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:16.114 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:02:15 smithi088 podman[153766]: 2023-07-14T19:02:16.462 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:02:16 smithi088 ceph-mds[153806]: -- 172.21.15.88:0/3810950010 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55925e7811e0 con 0x55925e757000 2023-07-14T19:02:16.463 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:02:16 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-h[153782]: starting mds.h at 2023-07-14T19:02:16.463 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:02:16 smithi088 bash[153766]: a7cfc0450c0986ef99e77d070f5caf59b6ec7259f0c4c4cdd683453a0d37c90a 2023-07-14T19:02:16.463 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:02:16 smithi088 systemd[1]: Started Ceph mds.h for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:16.602 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.738+0000 7faeb4f03700 1 -- 172.21.15.78:0/1859960451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 msgr2=0x7faeb0100570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.738+0000 7faeb4f03700 1 --2- 172.21.15.78:0/1859960451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 0x7faeb0100570 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7faea0009990 tx=0x7faea0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.738+0000 7faeb4f03700 1 -- 172.21.15.78:0/1859960451 shutdown_connections 2023-07-14T19:02:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.738+0000 7faeb4f03700 1 --2- 172.21.15.78:0/1859960451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb00ff760 0x7faeb0107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.743 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.738+0000 7faeb4f03700 1 --2- 172.21.15.78:0/1859960451 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 0x7faeb0100570 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.738+0000 7faeb4f03700 1 --2- 172.21.15.78:0/1859960451 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faeb00fedb0 0x7faeb00ff220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.739+0000 7faeb4f03700 1 -- 172.21.15.78:0/1859960451 >> 172.21.15.78:0/1859960451 conn(0x7faeb00f98d0 msgr2=0x7faeb00fbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:16.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.739+0000 7faeb4f03700 1 -- 172.21.15.78:0/1859960451 shutdown_connections 2023-07-14T19:02:16.744 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.739+0000 7faeb4f03700 1 -- 172.21.15.78:0/1859960451 wait complete. 2023-07-14T19:02:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.739+0000 7faeb4f03700 1 Processor -- start 2023-07-14T19:02:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.739+0000 7faeb4f03700 1 -- start start 2023-07-14T19:02:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeb4f03700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb00fedb0 0x7faeb0194740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:16.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeb4f03700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faeb00ff760 0x7faeb0194c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeb4f03700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 0x7faeb019bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeb4f03700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb019c2c0 con 0x7faeb0100160 2023-07-14T19:02:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeb4f03700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb019c400 con 0x7faeb00ff760 2023-07-14T19:02:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeb4f03700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb019c590 con 0x7faeb00fedb0 2023-07-14T19:02:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 0x7faeb019bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:16.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeadd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faeb00ff760 0x7faeb0194c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:16.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 0x7faeb019bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34278/0 (socket says 172.21.15.78:34278) 2023-07-14T19:02:16.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeadd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faeb00ff760 0x7faeb0194c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.88:3300/0 says I am v2:172.21.15.78:57698/0 (socket says 172.21.15.78:57698) 2023-07-14T19:02:16.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 -- 172.21.15.78:0/4150641162 learned_addr learned my addr 172.21.15.78:0/4150641162 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:16.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeae59c700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb00fedb0 0x7faeb0194740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:16.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 -- 172.21.15.78:0/4150641162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb00fedb0 msgr2=0x7faeb0194740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:16.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb00fedb0 0x7faeb0194740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 -- 172.21.15.78:0/4150641162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faeb00ff760 msgr2=0x7faeb0194c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:16.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faeb00ff760 0x7faeb0194c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 -- 172.21.15.78:0/4150641162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea00095f0 con 0x7faeb0100160 2023-07-14T19:02:16.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeadd9b700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faeb00ff760 0x7faeb0194c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:02:16.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeaed9d700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 0x7faeb019bcf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7faea800ada0 tx=0x7faea800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:16.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7fae9f7fe700 1 -- 172.21.15.78:0/4150641162 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea800f070 con 0x7faeb0100160 2023-07-14T19:02:16.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7fae9f7fe700 1 -- 172.21.15.78:0/4150641162 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7faea8007b30 con 0x7faeb0100160 2023-07-14T19:02:16.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb019c870 con 0x7faeb0100160 2023-07-14T19:02:16.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7fae9f7fe700 1 -- 172.21.15.78:0/4150641162 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea8005280 con 0x7faeb0100160 2023-07-14T19:02:16.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.740+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb019cd30 con 0x7faeb0100160 2023-07-14T19:02:16.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.741+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb010a1d0 con 0x7faeb0100160 2023-07-14T19:02:16.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.743+0000 7fae9f7fe700 1 -- 172.21.15.78:0/4150641162 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7faea800a090 con 0x7faeb0100160 2023-07-14T19:02:16.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.743+0000 7fae9f7fe700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fae980a11a0 0x7fae980a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:16.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.743+0000 7faeae59c700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fae980a11a0 0x7fae980a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:16.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.743+0000 7fae9f7fe700 1 -- 172.21.15.78:0/4150641162 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7faea80cbfb0 con 0x7faeb0100160 2023-07-14T19:02:16.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.744+0000 7faeae59c700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fae980a11a0 0x7fae980a3650 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faea000b5c0 tx=0x7faea0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:16.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.747+0000 7fae9f7fe700 1 -- 172.21.15.78:0/4150641162 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7faea8095c30 con 0x7faeb0100160 2023-07-14T19:02:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.976+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faeb0105b10 con 0x7faeb0100160 2023-07-14T19:02:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.976+0000 7fae9f7fe700 1 -- 172.21.15.78:0/4150641162 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6556 (secure 0 0 0) 0x7faea8099250 con 0x7faeb0100160 2023-07-14T19:02:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:16.981 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9}],"filesystems":[]} 2023-07-14T19:02:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fae980a11a0 msgr2=0x7fae980a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fae980a11a0 0x7fae980a3650 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7faea000b5c0 tx=0x7faea0005fb0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 msgr2=0x7faeb019bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 0x7faeb019bcf0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7faea800ada0 tx=0x7faea800c330 comp rx=0 tx=0).stop 2023-07-14T19:02:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 shutdown_connections 2023-07-14T19:02:16.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fae980a11a0 0x7fae980a3650 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7faeb0100160 0x7faeb019bcf0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faeb00fedb0 0x7faeb0194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 --2- 172.21.15.78:0/4150641162 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7faeb00ff760 0x7faeb0194c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 >> 172.21.15.78:0/4150641162 conn(0x7faeb00f98d0 msgr2=0x7faeb00fbba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 shutdown_connections 2023-07-14T19:02:16.984 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:16.979+0000 7faeb4f03700 1 -- 172.21.15.78:0/4150641162 wait complete. 2023-07-14T19:02:16.985 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 9 2023-07-14T19:02:17.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:17.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: Deploying daemon mds.i on smithi102 2023-07-14T19:02:17.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] up:boot 2023-07-14T19:02:17.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: fsmap 8 up:standby 2023-07-14T19:02:17.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:02:17.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:17 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/4150641162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:17.489 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:17.489 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.489 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.489 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.489 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:17.489 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:17.490 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:17.490 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: Deploying daemon mds.i on smithi102 2023-07-14T19:02:17.490 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] up:boot 2023-07-14T19:02:17.490 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: fsmap 8 up:standby 2023-07-14T19:02:17.490 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:02:17.490 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:17 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/4150641162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:17.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:17.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:17.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:17.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:17.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:17.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: Deploying daemon mds.i on smithi102 2023-07-14T19:02:17.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] up:boot 2023-07-14T19:02:17.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: fsmap 8 up:standby 2023-07-14T19:02:17.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:02:17.716 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:17 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/4150641162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:17.739 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:02:17 smithi102 systemd[1]: Starting Ceph mds.i for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:18.323 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:02:18 smithi102 podman[155454]: 2023-07-14T19:02:18.323 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:02:18 smithi102 ceph-mds[155494]: -- 172.21.15.102:0/3390612107 <== mon.1 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563b5c5af1e0 con 0x563b5c584c00 2023-07-14T19:02:18.324 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:02:18 smithi102 bash[155454]: 6e63d978d0ab27b9026fd21790c44ecc4adb5f843fa4807674247eaabd66fc83 2023-07-14T19:02:18.324 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:02:18 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-i[155470]: starting mds.i at 2023-07-14T19:02:18.630 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:02:18 smithi102 systemd[1]: Started Ceph mds.i for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:18.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:19.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 -- 172.21.15.78:0/693115563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0fe280 msgr2=0x7f8b6c0fe690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:19.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/693115563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0fe280 0x7f8b6c0fe690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8b6400b0a0 tx=0x7f8b6400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 -- 172.21.15.78:0/693115563 shutdown_connections 2023-07-14T19:02:19.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/693115563 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0ffe00 0x7f8b6c1066c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/693115563 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0fe280 0x7f8b6c0fe690 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/693115563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b6c0ff470 0x7f8b6c0ff8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 -- 172.21.15.78:0/693115563 >> 172.21.15.78:0/693115563 conn(0x7f8b6c0f98a0 msgr2=0x7f8b6c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:19.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 -- 172.21.15.78:0/693115563 shutdown_connections 2023-07-14T19:02:19.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.122+0000 7f8b72bb8700 1 -- 172.21.15.78:0/693115563 wait complete. 2023-07-14T19:02:19.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b72bb8700 1 Processor -- start 2023-07-14T19:02:19.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b72bb8700 1 -- start start 2023-07-14T19:02:19.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b72bb8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b6c0fe280 0x7f8b6c1947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:19.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b72bb8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0ff470 0x7f8b6c194d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:19.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b72bb8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0ffe00 0x7f8b6c19bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:19.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b72bb8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c19c360 con 0x7f8b6c0ff470 2023-07-14T19:02:19.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b72bb8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c19c4a0 con 0x7f8b6c0ffe00 2023-07-14T19:02:19.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b72bb8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6c19c630 con 0x7f8b6c0fe280 2023-07-14T19:02:19.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0ff470 0x7f8b6c194d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:19.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0ff470 0x7f8b6c194d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34290/0 (socket says 172.21.15.78:34290) 2023-07-14T19:02:19.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b6bfff700 1 -- 172.21.15.78:0/3791113771 learned_addr learned my addr 172.21.15.78:0/3791113771 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:19.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b70954700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b6c0fe280 0x7f8b6c1947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:19.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.123+0000 7f8b71155700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0ffe00 0x7f8b6c19bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:19.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b6bfff700 1 -- 172.21.15.78:0/3791113771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b6c0fe280 msgr2=0x7f8b6c1947e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:19.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b6bfff700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b6c0fe280 0x7f8b6c1947e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b6bfff700 1 -- 172.21.15.78:0/3791113771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0ffe00 msgr2=0x7f8b6c19bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:19.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b6bfff700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0ffe00 0x7f8b6c19bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b6bfff700 1 -- 172.21.15.78:0/3791113771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b64009d00 con 0x7f8b6c0ff470 2023-07-14T19:02:19.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b71155700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0ffe00 0x7f8b6c19bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:19.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b6bfff700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0ff470 0x7f8b6c194d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007b50 tx=0x7f8b5c007e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:19.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b69ffb700 1 -- 172.21.15.78:0/3791113771 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c011070 con 0x7f8b6c0ff470 2023-07-14T19:02:19.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b6c1009b0 con 0x7f8b6c0ff470 2023-07-14T19:02:19.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b69ffb700 1 -- 172.21.15.78:0/3791113771 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8b5c00c2e0 con 0x7f8b6c0ff470 2023-07-14T19:02:19.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b69ffb700 1 -- 172.21.15.78:0/3791113771 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c00a830 con 0x7f8b6c0ff470 2023-07-14T19:02:19.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.124+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b6c100ec0 con 0x7f8b6c0ff470 2023-07-14T19:02:19.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.125+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b6c109950 con 0x7f8b6c0ff470 2023-07-14T19:02:19.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.126+0000 7f8b69ffb700 1 -- 172.21.15.78:0/3791113771 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8b5c00e500 con 0x7f8b6c0ff470 2023-07-14T19:02:19.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.126+0000 7f8b69ffb700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b580a11a0 0x7f8b580a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:19.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.127+0000 7f8b70954700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b580a11a0 0x7f8b580a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:19.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.127+0000 7f8b69ffb700 1 -- 172.21.15.78:0/3791113771 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8b5c0cd750 con 0x7f8b6c0ff470 2023-07-14T19:02:19.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.127+0000 7f8b70954700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b580a11a0 0x7f8b580a3650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009fd0 tx=0x7f8b6400b7d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:19.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.131+0000 7f8b69ffb700 1 -- 172.21.15.78:0/3791113771 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8b5c097330 con 0x7f8b6c0ff470 2023-07-14T19:02:19.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.385+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b6c1061e0 con 0x7f8b6c0ff470 2023-07-14T19:02:19.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.385+0000 7f8b69ffb700 1 -- 172.21.15.78:0/3791113771 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7315 (secure 0 0 0) 0x7f8b5c00e7b0 con 0x7f8b6c0ff470 2023-07-14T19:02:19.390 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:19.391 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:02:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b580a11a0 msgr2=0x7f8b580a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b580a11a0 0x7f8b580a3650 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8b64009fd0 tx=0x7f8b6400b7d0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0ff470 msgr2=0x7f8b6c194d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0ff470 0x7f8b6c194d20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007b50 tx=0x7f8b5c007e60 comp rx=0 tx=0).stop 2023-07-14T19:02:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 shutdown_connections 2023-07-14T19:02:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b580a11a0 0x7f8b580a3650 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b6c0ffe00 0x7f8b6c19bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b6c0fe280 0x7f8b6c1947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 --2- 172.21.15.78:0/3791113771 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b6c0ff470 0x7f8b6c194d20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 >> 172.21.15.78:0/3791113771 conn(0x7f8b6c0f98a0 msgr2=0x7f8b6c066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 shutdown_connections 2023-07-14T19:02:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:19.387+0000 7f8b72bb8700 1 -- 172.21.15.78:0/3791113771 wait complete. 2023-07-14T19:02:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 10 2023-07-14T19:02:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:19.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:19.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:19.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: Deploying daemon mds.g on smithi078 2023-07-14T19:02:19.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] up:boot 2023-07-14T19:02:19.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: fsmap 9 up:standby 2023-07-14T19:02:19.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:19 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:02:19.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:19.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.691 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:19.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:19.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:19.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: Deploying daemon mds.g on smithi078 2023-07-14T19:02:19.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] up:boot 2023-07-14T19:02:19.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: fsmap 9 up:standby 2023-07-14T19:02:19.692 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:19 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:02:19.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:19.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:19.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:19.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:19.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:19.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: Deploying daemon mds.g on smithi078 2023-07-14T19:02:19.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] up:boot 2023-07-14T19:02:19.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: fsmap 9 up:standby 2023-07-14T19:02:19.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:19 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:02:20.048 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:02:19 smithi078 systemd[1]: Starting Ceph mds.g for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:20.547 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:02:20 smithi078 podman[161764]: 2023-07-14T19:02:20.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:20 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3791113771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:20.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:20 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3791113771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:20.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:20 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3791113771' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:21.047 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:02:20 smithi078 ceph-mds[161804]: -- 172.21.15.78:0/2833300182 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561f64e9b1e0 con 0x561f64e70800 2023-07-14T19:02:21.048 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:02:20 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-g[161780]: starting mds.g at 2023-07-14T19:02:21.048 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:02:20 smithi078 bash[161764]: 8a0c7d65f21bc12b6a4994fd88ddbde1120327a69fbbd7d96feeb9055c35ca48 2023-07-14T19:02:21.048 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:02:20 smithi078 systemd[1]: Started Ceph mds.g for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:21.406 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.530+0000 7ff05e321700 1 -- 172.21.15.78:0/873040678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff058100110 msgr2=0x7ff058100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:21.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.530+0000 7ff05e321700 1 --2- 172.21.15.78:0/873040678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff058100110 0x7ff058100520 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff048009990 tx=0x7ff048009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.530+0000 7ff05e321700 1 -- 172.21.15.78:0/873040678 shutdown_connections 2023-07-14T19:02:21.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.530+0000 7ff05e321700 1 --2- 172.21.15.78:0/873040678 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff0580ff710 0x7ff058106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.530+0000 7ff05e321700 1 --2- 172.21.15.78:0/873040678 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff058100110 0x7ff058100520 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.530+0000 7ff05e321700 1 --2- 172.21.15.78:0/873040678 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0580fed60 0x7ff0580ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.530+0000 7ff05e321700 1 -- 172.21.15.78:0/873040678 >> 172.21.15.78:0/873040678 conn(0x7ff0580f98a0 msgr2=0x7ff0580fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:21.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.530+0000 7ff05e321700 1 -- 172.21.15.78:0/873040678 shutdown_connections 2023-07-14T19:02:21.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 -- 172.21.15.78:0/873040678 wait complete. 2023-07-14T19:02:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 Processor -- start 2023-07-14T19:02:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 -- start start 2023-07-14T19:02:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0580fed60 0x7ff0580fe7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0580ff710 0x7ff05819a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff058100110 0x7ff05819cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05819d310 con 0x7ff0580fed60 2023-07-14T19:02:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05819d480 con 0x7ff0580ff710 2023-07-14T19:02:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.531+0000 7ff05e321700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05819d640 con 0x7ff058100110 2023-07-14T19:02:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0580fed60 0x7ff0580fe7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0580fed60 0x7ff0580fe7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34314/0 (socket says 172.21.15.78:34314) 2023-07-14T19:02:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 -- 172.21.15.78:0/1074435470 learned_addr learned my addr 172.21.15.78:0/1074435470 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:21.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff05c8be700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff058100110 0x7ff05819cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:21.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff0577fe700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0580ff710 0x7ff05819a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 -- 172.21.15.78:0/1074435470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff058100110 msgr2=0x7ff05819cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff058100110 0x7ff05819cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 -- 172.21.15.78:0/1074435470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0580ff710 msgr2=0x7ff05819a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0580ff710 0x7ff05819a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 -- 172.21.15.78:0/1074435470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0480095f0 con 0x7ff0580fed60 2023-07-14T19:02:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff0577fe700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0580ff710 0x7ff05819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff057fff700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0580fed60 0x7ff0580fe7e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff0480059d0 tx=0x7ff048003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff0557fa700 1 -- 172.21.15.78:0/1074435470 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff04801d070 con 0x7ff0580fed60 2023-07-14T19:02:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05819d8c0 con 0x7ff0580fed60 2023-07-14T19:02:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff0557fa700 1 -- 172.21.15.78:0/1074435470 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff04800bd70 con 0x7ff0580fed60 2023-07-14T19:02:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff0557fa700 1 -- 172.21.15.78:0/1074435470 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff048022e30 con 0x7ff0580fed60 2023-07-14T19:02:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.532+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05819dce0 con 0x7ff0580fed60 2023-07-14T19:02:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.533+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff058105b00 con 0x7ff0580fed60 2023-07-14T19:02:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.535+0000 7ff0557fa700 1 -- 172.21.15.78:0/1074435470 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff048003ff0 con 0x7ff0580fed60 2023-07-14T19:02:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.535+0000 7ff0557fa700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff0400a11a0 0x7ff0400a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:21.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.535+0000 7ff0577fe700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff0400a11a0 0x7ff0400a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:21.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.535+0000 7ff0557fa700 1 -- 172.21.15.78:0/1074435470 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff0480ce1a0 con 0x7ff0580fed60 2023-07-14T19:02:21.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.536+0000 7ff0577fe700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff0400a11a0 0x7ff0400a3650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff04c005fd0 tx=0x7ff04c005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:21.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.539+0000 7ff0557fa700 1 -- 172.21.15.78:0/1074435470 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff048097e20 con 0x7ff0580fed60 2023-07-14T19:02:21.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:21 smithi102 ceph-mon[129280]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:21.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:21.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:21.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:21 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:21.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:21 smithi078 ceph-mon[127425]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:21.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.632 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:21.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:21.633 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:21 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:21.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:21 smithi088 ceph-mon[128973]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:21.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:21.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:21.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:21.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:21 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:21.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.783+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff05802c8e0 con 0x7ff0580fed60 2023-07-14T19:02:21.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.783+0000 7ff0557fa700 1 -- 172.21.15.78:0/1074435470 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8070 (secure 0 0 0) 0x7ff04809b440 con 0x7ff0580fed60 2023-07-14T19:02:21.792 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:21.793 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:02:21.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff0400a11a0 msgr2=0x7ff0400a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff0400a11a0 0x7ff0400a3650 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7ff04c005fd0 tx=0x7ff04c005dc0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0580fed60 msgr2=0x7ff0580fe7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0580fed60 0x7ff0580fe7e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff0480059d0 tx=0x7ff048003b00 comp rx=0 tx=0).stop 2023-07-14T19:02:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 shutdown_connections 2023-07-14T19:02:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff0400a11a0 0x7ff0400a3650 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff058100110 0x7ff05819cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff0580fed60 0x7ff0580fe7e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 --2- 172.21.15.78:0/1074435470 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff0580ff710 0x7ff05819a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:21.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 >> 172.21.15.78:0/1074435470 conn(0x7ff0580f98a0 msgr2=0x7ff0580fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:21.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 shutdown_connections 2023-07-14T19:02:21.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:21.786+0000 7ff05e321700 1 -- 172.21.15.78:0/1074435470 wait complete. 2023-07-14T19:02:21.795 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 11 2023-07-14T19:02:22.298 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:02:22 smithi078 systemd[1]: Starting Ceph mds.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:22.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:22 smithi088 ceph-mon[128973]: Deploying daemon mds.a on smithi078 2023-07-14T19:02:22.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:22 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.78:6838/680498429,v1:172.21.15.78:6839/680498429] up:boot 2023-07-14T19:02:22.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:22 smithi088 ceph-mon[128973]: fsmap 10 up:standby 2023-07-14T19:02:22.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:22 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:02:22.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:22 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1074435470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:22.738 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:02:22 smithi078 podman[162149]: 2023-07-14T19:02:22.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:22 smithi078 ceph-mon[127425]: Deploying daemon mds.a on smithi078 2023-07-14T19:02:22.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:22 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.78:6838/680498429,v1:172.21.15.78:6839/680498429] up:boot 2023-07-14T19:02:22.738 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:22 smithi078 ceph-mon[127425]: fsmap 10 up:standby 2023-07-14T19:02:22.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:22 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:02:22.739 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:22 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1074435470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:22.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:22 smithi102 ceph-mon[129280]: Deploying daemon mds.a on smithi078 2023-07-14T19:02:22.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:22 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.78:6838/680498429,v1:172.21.15.78:6839/680498429] up:boot 2023-07-14T19:02:22.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:22 smithi102 ceph-mon[129280]: fsmap 10 up:standby 2023-07-14T19:02:22.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:22 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2023-07-14T19:02:22.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:22 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1074435470' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:23.047 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:02:22 smithi078 bash[162149]: 196f8f1f3ef2f469e13e313b8a33ca6e9543e01ef2ddc1e6d3fe361278ea8c49 2023-07-14T19:02:23.048 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:02:22 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-a[162165]: starting mds.a at 2023-07-14T19:02:23.048 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:02:22 smithi078 systemd[1]: Started Ceph mds.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:23.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:23 smithi088 ceph-mon[128973]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:23.675 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.676 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:23.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:23.677 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:23 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:23.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:23 smithi078 ceph-mon[127425]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:23.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:23.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:23.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:23 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:23.803 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:23.880 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:23 smithi102 ceph-mon[129280]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:23.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:23.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2023-07-14T19:02:23.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2023-07-14T19:02:23.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:23 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:23.947 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 -- 172.21.15.78:0/3040925042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 msgr2=0x7f34b4106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/3040925042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 0x7f34b4106f40 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00b0a0 tx=0x7f34ac00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 -- 172.21.15.78:0/3040925042 shutdown_connections 2023-07-14T19:02:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/3040925042 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 0x7f34b4106f40 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/3040925042 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34b40feb00 0x7f34b40fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:23.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/3040925042 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b40ffcf0 0x7f34b4100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 -- 172.21.15.78:0/3040925042 >> 172.21.15.78:0/3040925042 conn(0x7f34b40fa100 msgr2=0x7f34b40fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 -- 172.21.15.78:0/3040925042 shutdown_connections 2023-07-14T19:02:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.943+0000 7f34b9c9c700 1 -- 172.21.15.78:0/3040925042 wait complete. 2023-07-14T19:02:23.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.944+0000 7f34b9c9c700 1 Processor -- start 2023-07-14T19:02:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.944+0000 7f34b9c9c700 1 -- start start 2023-07-14T19:02:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.944+0000 7f34b9c9c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b40feb00 0x7f34b4195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.944+0000 7f34b9c9c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34b40ffcf0 0x7f34b4195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.944+0000 7f34b9c9c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 0x7f34b419ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:23.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.944+0000 7f34b9c9c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b419d480 con 0x7f34b4100680 2023-07-14T19:02:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.944+0000 7f34b9c9c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b419d5c0 con 0x7f34b40feb00 2023-07-14T19:02:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.944+0000 7f34b9c9c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b419d750 con 0x7f34b40ffcf0 2023-07-14T19:02:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 0x7f34b419ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b2ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34b40ffcf0 0x7f34b4195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 0x7f34b419ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34344/0 (socket says 172.21.15.78:34344) 2023-07-14T19:02:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 -- 172.21.15.78:0/1760284627 learned_addr learned my addr 172.21.15.78:0/1760284627 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b37fe700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b40feb00 0x7f34b4195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:23.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 -- 172.21.15.78:0/1760284627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34b40ffcf0 msgr2=0x7f34b4195e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:23.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34b40ffcf0 0x7f34b4195e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:23.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 -- 172.21.15.78:0/1760284627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b40feb00 msgr2=0x7f34b4195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:23.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b40feb00 0x7f34b4195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:23.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 -- 172.21.15.78:0/1760284627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34ac009d00 con 0x7f34b4100680 2023-07-14T19:02:23.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b3fff700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 0x7f34b419ceb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00b830 tx=0x7f34ac004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:23.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b0ff9700 1 -- 172.21.15.78:0/1760284627 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac00e040 con 0x7f34b4100680 2023-07-14T19:02:23.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b0ff9700 1 -- 172.21.15.78:0/1760284627 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f34ac009440 con 0x7f34b4100680 2023-07-14T19:02:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b0ff9700 1 -- 172.21.15.78:0/1760284627 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34ac01db60 con 0x7f34b4100680 2023-07-14T19:02:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b419d9d0 con 0x7f34b4100680 2023-07-14T19:02:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.945+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b419ddf0 con 0x7f34b4100680 2023-07-14T19:02:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.946+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34b4106b50 con 0x7f34b4100680 2023-07-14T19:02:23.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.947+0000 7f34b0ff9700 1 -- 172.21.15.78:0/1760284627 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f34ac005450 con 0x7f34b4100680 2023-07-14T19:02:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.948+0000 7f34b0ff9700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34a00a1220 0x7f34a00a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.948+0000 7f34b0ff9700 1 -- 172.21.15.78:0/1760284627 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f34ac0cdb90 con 0x7f34b4100680 2023-07-14T19:02:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.948+0000 7f34b37fe700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34a00a1220 0x7f34a00a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.948+0000 7f34b37fe700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34a00a1220 0x7f34a00a36d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f34b40ffb50 tx=0x7f349c00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:23.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:23.952+0000 7f34b0ff9700 1 -- 172.21.15.78:0/1760284627 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f34ac097810 con 0x7f34b4100680 2023-07-14T19:02:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.184+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f34b402cd00 con 0x7f34b4100680 2023-07-14T19:02:24.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.185+0000 7f34b0ff9700 1 -- 172.21.15.78:0/1760284627 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+8819 (secure 0 0 0) 0x7f34ac005700 con 0x7f34b4100680 2023-07-14T19:02:24.189 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:24.189 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[]} 2023-07-14T19:02:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34a00a1220 msgr2=0x7f34a00a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:24.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34a00a1220 0x7f34a00a36d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f34b40ffb50 tx=0x7f349c00a380 comp rx=0 tx=0).stop 2023-07-14T19:02:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 msgr2=0x7f34b419ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 0x7f34b419ceb0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f34ac00b830 tx=0x7f34ac004ce0 comp rx=0 tx=0).stop 2023-07-14T19:02:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 shutdown_connections 2023-07-14T19:02:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34a00a1220 0x7f34a00a36d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f34b40feb00 0x7f34b4195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f34b4100680 0x7f34b419ceb0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:24.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 --2- 172.21.15.78:0/1760284627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f34b40ffcf0 0x7f34b4195e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 >> 172.21.15.78:0/1760284627 conn(0x7f34b40fa100 msgr2=0x7f34b40fc2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 shutdown_connections 2023-07-14T19:02:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:24.187+0000 7f34b9c9c700 1 -- 172.21.15.78:0/1760284627 wait complete. 2023-07-14T19:02:24.193 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 12 2023-07-14T19:02:24.364 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:02:24 smithi088 systemd[1]: Starting Ceph mds.b for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:02:24.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:24 smithi078 ceph-mon[127425]: Deploying daemon mds.b on smithi088 2023-07-14T19:02:24.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:24 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.78:6840/1070208,v1:172.21.15.78:6841/1070208] up:boot 2023-07-14T19:02:24.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:24 smithi078 ceph-mon[127425]: fsmap 11 up:standby 2023-07-14T19:02:24.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:24 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:02:24.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:24 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1760284627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:24.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:24 smithi088 ceph-mon[128973]: Deploying daemon mds.b on smithi088 2023-07-14T19:02:24.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:24 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.78:6840/1070208,v1:172.21.15.78:6841/1070208] up:boot 2023-07-14T19:02:24.684 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:24 smithi088 ceph-mon[128973]: fsmap 11 up:standby 2023-07-14T19:02:24.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:24 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:02:24.685 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:24 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1760284627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:24.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:24 smithi102 ceph-mon[129280]: Deploying daemon mds.b on smithi088 2023-07-14T19:02:24.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:24 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.78:6840/1070208,v1:172.21.15.78:6841/1070208] up:boot 2023-07-14T19:02:24.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:24 smithi102 ceph-mon[129280]: fsmap 11 up:standby 2023-07-14T19:02:24.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:24 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2023-07-14T19:02:24.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:24 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1760284627' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:24.951 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:02:24 smithi088 podman[154106]: 2023-07-14T19:02:24.951 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:02:24 smithi088 ceph-mds[154146]: -- 172.21.15.88:0/1163621946 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5619526df1e0 con 0x5619526b5000 2023-07-14T19:02:24.952 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:02:24 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b[154122]: starting mds.b at 2023-07-14T19:02:24.952 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:02:24 smithi088 bash[154106]: b4f6c42092d1b607df68850c7ed69a3f34000af6f7643e5a89a812f07ed72986 2023-07-14T19:02:25.212 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:02:24 smithi088 systemd[1]: Started Ceph mds.b for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:02:25.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:25 smithi088 ceph-mon[128973]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:25.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:25 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:02:25.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:25 smithi078 ceph-mon[127425]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:25.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:25 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:02:25.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:25 smithi102 ceph-mon[129280]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:25.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:25.884 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:25 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:02:26.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:26.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 -- 172.21.15.78:0/2436702377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 msgr2=0x7f27d80fe2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 --2- 172.21.15.78:0/2436702377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 0x7f27d80fe2d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f27c8009990 tx=0x7f27c8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 -- 172.21.15.78:0/2436702377 shutdown_connections 2023-07-14T19:02:26.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 --2- 172.21.15.78:0/2436702377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27d8101820 0x7f27d81080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 --2- 172.21.15.78:0/2436702377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 0x7f27d80fe2d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 --2- 172.21.15.78:0/2436702377 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27d8100e90 0x7f27d81012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 -- 172.21.15.78:0/2436702377 >> 172.21.15.78:0/2436702377 conn(0x7f27d80f98a0 msgr2=0x7f27d80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:26.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 -- 172.21.15.78:0/2436702377 shutdown_connections 2023-07-14T19:02:26.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.341+0000 7f27dc978700 1 -- 172.21.15.78:0/2436702377 wait complete. 2023-07-14T19:02:26.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27dc978700 1 Processor -- start 2023-07-14T19:02:26.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27dc978700 1 -- start start 2023-07-14T19:02:26.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27dc978700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 0x7f27d81958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27dc978700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27d8100e90 0x7f27d8195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27dc978700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27d8101820 0x7f27d819ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27dc978700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d819d450 con 0x7f27d80fdec0 2023-07-14T19:02:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27dc978700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d819d590 con 0x7f27d8101820 2023-07-14T19:02:26.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27dc978700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d819d720 con 0x7f27d8100e90 2023-07-14T19:02:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 0x7f27d81958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 0x7f27d81958d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34354/0 (socket says 172.21.15.78:34354) 2023-07-14T19:02:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d659c700 1 -- 172.21.15.78:0/858289853 learned_addr learned my addr 172.21.15.78:0/858289853 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d5d9b700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27d8100e90 0x7f27d8195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d6d9d700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27d8101820 0x7f27d819ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d659c700 1 -- 172.21.15.78:0/858289853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27d8100e90 msgr2=0x7f27d8195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d659c700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27d8100e90 0x7f27d8195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d659c700 1 -- 172.21.15.78:0/858289853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27d8101820 msgr2=0x7f27d819ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d659c700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27d8101820 0x7f27d819ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.342+0000 7f27d659c700 1 -- 172.21.15.78:0/858289853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c80095f0 con 0x7f27d80fdec0 2023-07-14T19:02:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.343+0000 7f27d659c700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 0x7f27d81958d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f27c80098e0 tx=0x7f27c8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.343+0000 7f27c77fe700 1 -- 172.21.15.78:0/858289853 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c801d070 con 0x7f27d80fdec0 2023-07-14T19:02:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.343+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d819d9a0 con 0x7f27d80fdec0 2023-07-14T19:02:26.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.343+0000 7f27c77fe700 1 -- 172.21.15.78:0/858289853 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f27c800bdb0 con 0x7f27d80fdec0 2023-07-14T19:02:26.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.343+0000 7f27c77fe700 1 -- 172.21.15.78:0/858289853 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27c8022cf0 con 0x7f27d80fdec0 2023-07-14T19:02:26.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.343+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d819de20 con 0x7f27d80fdec0 2023-07-14T19:02:26.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.344+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d8107280 con 0x7f27d80fdec0 2023-07-14T19:02:26.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.345+0000 7f27c77fe700 1 -- 172.21.15.78:0/858289853 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f27c8022470 con 0x7f27d80fdec0 2023-07-14T19:02:26.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.345+0000 7f27c77fe700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f27c00a11a0 0x7f27c00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:26.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.345+0000 7f27c77fe700 1 -- 172.21.15.78:0/858289853 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f27c80cdac0 con 0x7f27d80fdec0 2023-07-14T19:02:26.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.345+0000 7f27d5d9b700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f27c00a11a0 0x7f27c00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:26.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.346+0000 7f27d5d9b700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f27c00a11a0 0x7f27c00a3650 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009c60 tx=0x7f27cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:26.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.350+0000 7f27c77fe700 1 -- 172.21.15.78:0/858289853 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f27c8097740 con 0x7f27d80fdec0 2023-07-14T19:02:26.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:26 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] up:boot 2023-07-14T19:02:26.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:26 smithi078 ceph-mon[127425]: fsmap 12 up:standby 2023-07-14T19:02:26.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:26 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:02:26.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.600+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27d802cae0 con 0x7f27d80fdec0 2023-07-14T19:02:26.604 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.600+0000 7f27c77fe700 1 -- 172.21.15.78:0/858289853 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+9577 (secure 0 0 0) 0x7f27c809ad60 con 0x7f27d80fdec0 2023-07-14T19:02:26.604 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:26.605 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10},{"gid":24503,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":13}],"filesystems":[]} 2023-07-14T19:02:26.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f27c00a11a0 msgr2=0x7f27c00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f27c00a11a0 0x7f27c00a3650 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f27cc009c60 tx=0x7f27cc009260 comp rx=0 tx=0).stop 2023-07-14T19:02:26.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 msgr2=0x7f27d81958d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 0x7f27d81958d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f27c80098e0 tx=0x7f27c8004730 comp rx=0 tx=0).stop 2023-07-14T19:02:26.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 shutdown_connections 2023-07-14T19:02:26.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f27c00a11a0 0x7f27c00a3650 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f27d8101820 0x7f27d819ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f27d80fdec0 0x7f27d81958d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 --2- 172.21.15.78:0/858289853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27d8100e90 0x7f27d8195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 >> 172.21.15.78:0/858289853 conn(0x7f27d80f98a0 msgr2=0x7f27d81043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:26.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 shutdown_connections 2023-07-14T19:02:26.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.603+0000 7f27dc978700 1 -- 172.21.15.78:0/858289853 wait complete. 2023-07-14T19:02:26.609 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 13 2023-07-14T19:02:26.619 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:26.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:26 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] up:boot 2023-07-14T19:02:26.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:26 smithi088 ceph-mon[128973]: fsmap 12 up:standby 2023-07-14T19:02:26.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:26 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:02:26.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 -- 172.21.15.78:0/2536273246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280ffd30 msgr2=0x7fa828100180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/2536273246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280ffd30 0x7fa828100180 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa810009990 tx=0x7fa810009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 -- 172.21.15.78:0/2536273246 shutdown_connections 2023-07-14T19:02:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/2536273246 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8281006c0 0x7fa828106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/2536273246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8280feb40 0x7fa8280fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/2536273246 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280ffd30 0x7fa828100180 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 -- 172.21.15.78:0/2536273246 >> 172.21.15.78:0/2536273246 conn(0x7fa8280fa160 msgr2=0x7fa8280fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 -- 172.21.15.78:0/2536273246 shutdown_connections 2023-07-14T19:02:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.743+0000 7fa82ec6e700 1 -- 172.21.15.78:0/2536273246 wait complete. 2023-07-14T19:02:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 Processor -- start 2023-07-14T19:02:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 -- start start 2023-07-14T19:02:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280feb40 0x7fa828069300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8280ffd30 0x7fa82819a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:26.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8281006c0 0x7fa82819cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82819d3b0 con 0x7fa8280feb40 2023-07-14T19:02:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82819d520 con 0x7fa8280ffd30 2023-07-14T19:02:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82819d6e0 con 0x7fa8281006c0 2023-07-14T19:02:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280feb40 0x7fa828069300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280feb40 0x7fa828069300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34368/0 (socket says 172.21.15.78:34368) 2023-07-14T19:02:26.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 -- 172.21.15.78:0/1374385799 learned_addr learned my addr 172.21.15.78:0/1374385799 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa827fff700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8280ffd30 0x7fa82819a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82d20b700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8281006c0 0x7fa82819cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 -- 172.21.15.78:0/1374385799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8281006c0 msgr2=0x7fa82819cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8281006c0 0x7fa82819cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 -- 172.21.15.78:0/1374385799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8280ffd30 msgr2=0x7fa82819a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8280ffd30 0x7fa82819a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 -- 172.21.15.78:0/1374385799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8100095f0 con 0x7fa8280feb40 2023-07-14T19:02:26.752 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82d20b700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8281006c0 0x7fa82819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:26.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ca0a700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280feb40 0x7fa828069300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa82000efc0 tx=0x7fa82000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:26.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa825ffb700 1 -- 172.21.15.78:0/1374385799 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820018070 con 0x7fa8280feb40 2023-07-14T19:02:26.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa825ffb700 1 -- 172.21.15.78:0/1374385799 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa82000f040 con 0x7fa8280feb40 2023-07-14T19:02:26.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82819d960 con 0x7fa8280feb40 2023-07-14T19:02:26.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.744+0000 7fa825ffb700 1 -- 172.21.15.78:0/1374385799 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa820004280 con 0x7fa8280feb40 2023-07-14T19:02:26.753 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.745+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82819de40 con 0x7fa8280feb40 2023-07-14T19:02:26.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.746+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa828106080 con 0x7fa8280feb40 2023-07-14T19:02:26.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.747+0000 7fa825ffb700 1 -- 172.21.15.78:0/1374385799 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa82000ca70 con 0x7fa8280feb40 2023-07-14T19:02:26.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.747+0000 7fa825ffb700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa8140a11a0 0x7fa8140a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:26.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.747+0000 7fa827fff700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa8140a11a0 0x7fa8140a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:26.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.747+0000 7fa825ffb700 1 -- 172.21.15.78:0/1374385799 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa8200cc430 con 0x7fa8280feb40 2023-07-14T19:02:26.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.748+0000 7fa827fff700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa8140a11a0 0x7fa8140a3650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa810006010 tx=0x7fa81000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:26.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.752+0000 7fa825ffb700 1 -- 172.21.15.78:0/1374385799 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa8200960b0 con 0x7fa8280feb40 2023-07-14T19:02:26.880 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:26 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] up:boot 2023-07-14T19:02:26.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:26 smithi102 ceph-mon[129280]: fsmap 12 up:standby 2023-07-14T19:02:26.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:26 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:02:26.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.951+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa82802cb30 con 0x7fa8280feb40 2023-07-14T19:02:26.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.952+0000 7fa825ffb700 1 -- 172.21.15.78:0/1374385799 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+18384 (secure 0 0 0) 0x7fa8200996d0 con 0x7fa8280feb40 2023-07-14T19:02:26.956 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:26.956 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":77,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:01:50.722377+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":72,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa8140a11a0 msgr2=0x7fa8140a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa8140a11a0 0x7fa8140a3650 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa810006010 tx=0x7fa81000b260 comp rx=0 tx=0).stop 2023-07-14T19:02:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280feb40 msgr2=0x7fa828069300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:26.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280feb40 0x7fa828069300 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa82000efc0 tx=0x7fa82000c3a0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 shutdown_connections 2023-07-14T19:02:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa8140a11a0 0x7fa8140a3650 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8281006c0 0x7fa82819cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa8280feb40 0x7fa828069300 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 --2- 172.21.15.78:0/1374385799 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa8280ffd30 0x7fa82819a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 >> 172.21.15.78:0/1374385799 conn(0x7fa8280fa160 msgr2=0x7fa8281031c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:26.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 shutdown_connections 2023-07-14T19:02:26.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:26.954+0000 7fa82ec6e700 1 -- 172.21.15.78:0/1374385799 wait complete. 2023-07-14T19:02:26.971 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-07-14T19:02:26.972 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:02:27.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.095+0000 7f53de611700 1 -- 172.21.15.78:0/2695743446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80ffd10 msgr2=0x7f53d8100160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.095+0000 7f53de611700 1 --2- 172.21.15.78:0/2695743446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80ffd10 0x7f53d8100160 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f53cc009990 tx=0x7f53cc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 -- 172.21.15.78:0/2695743446 shutdown_connections 2023-07-14T19:02:27.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 --2- 172.21.15.78:0/2695743446 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d80feb20 0x7f53d80fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 --2- 172.21.15.78:0/2695743446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53d81006a0 0x7f53d8106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.100 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 --2- 172.21.15.78:0/2695743446 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80ffd10 0x7f53d8100160 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 -- 172.21.15.78:0/2695743446 >> 172.21.15.78:0/2695743446 conn(0x7f53d80fa160 msgr2=0x7f53d80fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:27.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 -- 172.21.15.78:0/2695743446 shutdown_connections 2023-07-14T19:02:27.101 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 -- 172.21.15.78:0/2695743446 wait complete. 2023-07-14T19:02:27.102 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 Processor -- start 2023-07-14T19:02:27.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 -- start start 2023-07-14T19:02:27.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80feb20 0x7f53d8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.096+0000 7f53de611700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53d80ffd10 0x7f53d819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53de611700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d81006a0 0x7f53d819cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53de611700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d819d400 con 0x7f53d80feb20 2023-07-14T19:02:27.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53de611700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d819d570 con 0x7f53d81006a0 2023-07-14T19:02:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53de611700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53d819d730 con 0x7f53d80ffd10 2023-07-14T19:02:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80feb20 0x7f53d8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80feb20 0x7f53d8069340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34382/0 (socket says 172.21.15.78:34382) 2023-07-14T19:02:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 -- 172.21.15.78:0/2249613395 learned_addr learned my addr 172.21.15.78:0/2249613395 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d77fe700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53d80ffd10 0x7f53d819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53dcbae700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d81006a0 0x7f53d819cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 -- 172.21.15.78:0/2249613395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53d80ffd10 msgr2=0x7f53d819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53d80ffd10 0x7f53d819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 -- 172.21.15.78:0/2249613395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d81006a0 msgr2=0x7f53d819cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d81006a0 0x7f53d819cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 -- 172.21.15.78:0/2249613395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c8009520 con 0x7f53d80feb20 2023-07-14T19:02:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d77fe700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53d80ffd10 0x7f53d819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:02:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d7fff700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80feb20 0x7f53d8069340 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f53c800ea50 tx=0x7f53c800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d57fa700 1 -- 172.21.15.78:0/2249613395 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c800cbd0 con 0x7f53d80feb20 2023-07-14T19:02:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d57fa700 1 -- 172.21.15.78:0/2249613395 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f53c800cd30 con 0x7f53d80feb20 2023-07-14T19:02:27.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53d57fa700 1 -- 172.21.15.78:0/2249613395 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c8018680 con 0x7f53d80feb20 2023-07-14T19:02:27.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53cc0095f0 con 0x7f53d80feb20 2023-07-14T19:02:27.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.097+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d819dd10 con 0x7f53d80feb20 2023-07-14T19:02:27.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.098+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d810a1a0 con 0x7f53d80feb20 2023-07-14T19:02:27.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.100+0000 7f53d57fa700 1 -- 172.21.15.78:0/2249613395 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f53c8010430 con 0x7f53d80feb20 2023-07-14T19:02:27.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.100+0000 7f53d57fa700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53c00a1150 0x7f53c00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.100+0000 7f53d77fe700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53c00a1150 0x7f53c00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.100+0000 7f53d57fa700 1 -- 172.21.15.78:0/2249613395 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f53c8014070 con 0x7f53d80feb20 2023-07-14T19:02:27.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.101+0000 7f53d77fe700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53c00a1150 0x7f53c00a3600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00b2e0 tx=0x7f53cc004500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:27.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.104+0000 7f53d57fa700 1 -- 172.21.15.78:0/2249613395 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f53c8096b70 con 0x7f53d80feb20 2023-07-14T19:02:27.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.300+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f53d8105aa0 con 0x7f53d80feb20 2023-07-14T19:02:27.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.300+0000 7f53d57fa700 1 -- 172.21.15.78:0/2249613395 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v77) v1 ==== 93+0+10 (secure 0 0 0) 0x7f53c80b8e60 con 0x7f53d80feb20 2023-07-14T19:02:27.305 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:02:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.302+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53c00a1150 msgr2=0x7f53c00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.302+0000 7f53de611700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53c00a1150 0x7f53c00a3600 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f53cc00b2e0 tx=0x7f53cc004500 comp rx=0 tx=0).stop 2023-07-14T19:02:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.302+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80feb20 msgr2=0x7f53d8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.302+0000 7f53de611700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80feb20 0x7f53d8069340 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f53c800ea50 tx=0x7f53c800ed60 comp rx=0 tx=0).stop 2023-07-14T19:02:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.303+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 shutdown_connections 2023-07-14T19:02:27.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.303+0000 7f53de611700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53c00a1150 0x7f53c00a3600 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.303+0000 7f53de611700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f53d81006a0 0x7f53d819cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.303+0000 7f53de611700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f53d80feb20 0x7f53d8069340 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.303+0000 7f53de611700 1 --2- 172.21.15.78:0/2249613395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f53d80ffd10 0x7f53d819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.303+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 >> 172.21.15.78:0/2249613395 conn(0x7f53d80fa160 msgr2=0x7f53d80fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.303+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 shutdown_connections 2023-07-14T19:02:27.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.303+0000 7f53de611700 1 -- 172.21.15.78:0/2249613395 wait complete. 2023-07-14T19:02:27.318 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-07-14T19:02:27.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 -- 172.21.15.78:0/882744616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 msgr2=0x7f658c108070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 --2- 172.21.15.78:0/882744616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 0x7f658c108070 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f658400b0a0 tx=0x7f658400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 -- 172.21.15.78:0/882744616 shutdown_connections 2023-07-14T19:02:27.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 --2- 172.21.15.78:0/882744616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 0x7f658c108070 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 --2- 172.21.15.78:0/882744616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f658c0ffc30 0x7f658c100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 --2- 172.21.15.78:0/882744616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f658c100e20 0x7f658c101270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 -- 172.21.15.78:0/882744616 >> 172.21.15.78:0/882744616 conn(0x7f658c0fb230 msgr2=0x7f658c0fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:27.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 -- 172.21.15.78:0/882744616 shutdown_connections 2023-07-14T19:02:27.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.439+0000 7f6593e34700 1 -- 172.21.15.78:0/882744616 wait complete. 2023-07-14T19:02:27.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6593e34700 1 Processor -- start 2023-07-14T19:02:27.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6593e34700 1 -- start start 2023-07-14T19:02:27.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6593e34700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f658c0ffc30 0x7f658c1958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6593e34700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f658c100e20 0x7f658c195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6593e34700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 0x7f658c19cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6593e34700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c19d470 con 0x7f658c1017b0 2023-07-14T19:02:27.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6593e34700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c19d5b0 con 0x7f658c100e20 2023-07-14T19:02:27.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6593e34700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f658c19d740 con 0x7f658c0ffc30 2023-07-14T19:02:27.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f65923d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 0x7f658c19cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f65923d1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 0x7f658c19cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34398/0 (socket says 172.21.15.78:34398) 2023-07-14T19:02:27.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f65923d1700 1 -- 172.21.15.78:0/2647859282 learned_addr learned my addr 172.21.15.78:0/2647859282 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:27.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f6591bd0700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f658c0ffc30 0x7f658c1958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.440+0000 7f65913cf700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f658c100e20 0x7f658c195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f65923d1700 1 -- 172.21.15.78:0/2647859282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f658c0ffc30 msgr2=0x7f658c1958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f65923d1700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f658c0ffc30 0x7f658c1958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f65923d1700 1 -- 172.21.15.78:0/2647859282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f658c100e20 msgr2=0x7f658c195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f65923d1700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f658c100e20 0x7f658c195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f65923d1700 1 -- 172.21.15.78:0/2647859282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6584009d00 con 0x7f658c1017b0 2023-07-14T19:02:27.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f6591bd0700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f658c0ffc30 0x7f658c1958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:02:27.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f65923d1700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 0x7f658c19cea0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f658400bad0 tx=0x7f658400bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:27.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f6582ffd700 1 -- 172.21.15.78:0/2647859282 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f658400e040 con 0x7f658c1017b0 2023-07-14T19:02:27.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f6582ffd700 1 -- 172.21.15.78:0/2647859282 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6584003c00 con 0x7f658c1017b0 2023-07-14T19:02:27.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f658c19d9c0 con 0x7f658c1017b0 2023-07-14T19:02:27.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f6582ffd700 1 -- 172.21.15.78:0/2647859282 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65840052a0 con 0x7f658c1017b0 2023-07-14T19:02:27.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.441+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f658c19de90 con 0x7f658c1017b0 2023-07-14T19:02:27.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.442+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f658c10b2c0 con 0x7f658c1017b0 2023-07-14T19:02:27.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.443+0000 7f6582ffd700 1 -- 172.21.15.78:0/2647859282 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f658401d470 con 0x7f658c1017b0 2023-07-14T19:02:27.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.444+0000 7f6582ffd700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f65780a1220 0x7f65780a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.444+0000 7f6591bd0700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f65780a1220 0x7f65780a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.444+0000 7f6582ffd700 1 -- 172.21.15.78:0/2647859282 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f65840cde60 con 0x7f658c1017b0 2023-07-14T19:02:27.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.444+0000 7f6591bd0700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f65780a1220 0x7f65780a36d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f658c100c80 tx=0x7f6588009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:27.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.448+0000 7f6582ffd700 1 -- 172.21.15.78:0/2647859282 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6584097ae0 con 0x7f658c1017b0 2023-07-14T19:02:27.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.657+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f658c10b4b0 con 0x7f658c1017b0 2023-07-14T19:02:27.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.658+0000 7f6582ffd700 1 -- 172.21.15.78:0/2647859282 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+65 (secure 0 0 0) 0x7f658409b100 con 0x7f658c1017b0 2023-07-14T19:02:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:27.662 INFO:teuthology.orchestra.run.smithi078.stdout:[ 2023-07-14T19:02:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: { 2023-07-14T19:02:27.662 INFO:teuthology.orchestra.run.smithi078.stdout: "poolnum": 1, 2023-07-14T19:02:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: "poolname": ".mgr" 2023-07-14T19:02:27.663 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-14T19:02:27.663 INFO:teuthology.orchestra.run.smithi078.stdout:] 2023-07-14T19:02:27.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.659+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f65780a1220 msgr2=0x7f65780a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f65780a1220 0x7f65780a36d0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f658c100c80 tx=0x7f6588009260 comp rx=0 tx=0).stop 2023-07-14T19:02:27.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 msgr2=0x7f658c19cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 0x7f658c19cea0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f658400bad0 tx=0x7f658400bb00 comp rx=0 tx=0).stop 2023-07-14T19:02:27.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 shutdown_connections 2023-07-14T19:02:27.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f65780a1220 0x7f65780a36d0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f658c1017b0 0x7f658c19cea0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f658c0ffc30 0x7f658c1958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 --2- 172.21.15.78:0/2647859282 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f658c100e20 0x7f658c195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 >> 172.21.15.78:0/2647859282 conn(0x7f658c0fb230 msgr2=0x7f658c0fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:27.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 shutdown_connections 2023-07-14T19:02:27.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.660+0000 7f6593e34700 1 -- 172.21.15.78:0/2647859282 wait complete. 2023-07-14T19:02:27.674 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:27 smithi078 ceph-mon[127425]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:27.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:27 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/858289853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:27.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:27 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1374385799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:27.675 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:27 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2249613395' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:27.676 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-07-14T19:02:27.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-07-14T19:02:27.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:27 smithi088 ceph-mon[128973]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:27.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:27 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/858289853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:27.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:27 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1374385799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:27.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:27 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2249613395' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 -- 172.21.15.78:0/228298084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80ff7f0 msgr2=0x7f4ee80ffc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/228298084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80ff7f0 0x7f4ee80ffc60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4ee000b0a0 tx=0x7f4ee000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 -- 172.21.15.78:0/228298084 shutdown_connections 2023-07-14T19:02:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/228298084 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80ff7f0 0x7f4ee80ffc60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/228298084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee80fe4f0 0x7f4ee80fe900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/228298084 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ee80fee40 0x7f4ee80ff2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 -- 172.21.15.78:0/228298084 >> 172.21.15.78:0/228298084 conn(0x7f4ee80fa160 msgr2=0x7f4ee80fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 -- 172.21.15.78:0/228298084 shutdown_connections 2023-07-14T19:02:27.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.804+0000 7f4eef3e8700 1 -- 172.21.15.78:0/228298084 wait complete. 2023-07-14T19:02:27.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eef3e8700 1 Processor -- start 2023-07-14T19:02:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eef3e8700 1 -- start start 2023-07-14T19:02:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eef3e8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ee80fe4f0 0x7f4ee8197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eef3e8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80fee40 0x7f4ee8198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eef3e8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee80ff7f0 0x7f4ee819f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eef3e8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee819f750 con 0x7f4ee80fee40 2023-07-14T19:02:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eef3e8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee819f890 con 0x7f4ee80fe4f0 2023-07-14T19:02:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eef3e8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee819fa20 con 0x7f4ee80ff7f0 2023-07-14T19:02:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eec983700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80fee40 0x7f4ee8198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eec983700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80fee40 0x7f4ee8198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34410/0 (socket says 172.21.15.78:34410) 2023-07-14T19:02:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eed184700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ee80fe4f0 0x7f4ee8197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eec983700 1 -- 172.21.15.78:0/692709351 learned_addr learned my addr 172.21.15.78:0/692709351 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eed985700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee80ff7f0 0x7f4ee819f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eec983700 1 -- 172.21.15.78:0/692709351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee80ff7f0 msgr2=0x7f4ee819f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eec983700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee80ff7f0 0x7f4ee819f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eec983700 1 -- 172.21.15.78:0/692709351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ee80fe4f0 msgr2=0x7f4ee8197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eec983700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ee80fe4f0 0x7f4ee8197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eec983700 1 -- 172.21.15.78:0/692709351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee0009d00 con 0x7f4ee80fee40 2023-07-14T19:02:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.805+0000 7f4eed184700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ee80fe4f0 0x7f4ee8197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:02:27.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.806+0000 7f4eec983700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80fee40 0x7f4ee8198110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800b530 tx=0x7f4ed800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.807+0000 7f4ede7fc700 1 -- 172.21.15.78:0/692709351 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8011830 con 0x7f4ee80fee40 2023-07-14T19:02:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.807+0000 7f4ede7fc700 1 -- 172.21.15.78:0/692709351 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4ed8011e20 con 0x7f4ee80fee40 2023-07-14T19:02:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.807+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee819fd00 con 0x7f4ee80fee40 2023-07-14T19:02:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.807+0000 7f4ede7fc700 1 -- 172.21.15.78:0/692709351 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed8010db0 con 0x7f4ee80fee40 2023-07-14T19:02:27.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.807+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee81a0280 con 0x7f4ee80fee40 2023-07-14T19:02:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.808+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee81092d0 con 0x7f4ee80fee40 2023-07-14T19:02:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.809+0000 7f4ede7fc700 1 -- 172.21.15.78:0/692709351 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4ed8011990 con 0x7f4ee80fee40 2023-07-14T19:02:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.809+0000 7f4ede7fc700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ed40a11d0 0x7f4ed40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:27.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.810+0000 7f4eed184700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ed40a11d0 0x7f4ed40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.810+0000 7f4ede7fc700 1 -- 172.21.15.78:0/692709351 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4ed80cc190 con 0x7f4ee80fee40 2023-07-14T19:02:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.810+0000 7f4eed184700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ed40a11d0 0x7f4ed40a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4005950 tx=0x7f4ee400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:27.814+0000 7f4ede7fc700 1 -- 172.21.15.78:0/692709351 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4ed8095d80 con 0x7f4ee80fee40 2023-07-14T19:02:27.880 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:27 smithi102 ceph-mon[129280]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:27.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:27 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/858289853' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:27.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:27 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1374385799' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:27.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:27 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2249613395' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:28.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.032+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4ee802c8e0 con 0x7f4ee80fee40 2023-07-14T19:02:28.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2647859282' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:02:28.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/692709351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:02:28.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:28.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:02:28.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:28 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2647859282' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:02:28.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/692709351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:02:28.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:28.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:02:28.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:28 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2647859282' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2023-07-14T19:02:28.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/692709351' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2023-07-14T19:02:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:02:28.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:28 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:28.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.921+0000 7f4ede7fc700 1 -- 172.21.15.78:0/692709351 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v78) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4ed80993a0 con 0x7f4ee80fee40 2023-07-14T19:02:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ed40a11d0 msgr2=0x7f4ed40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ed40a11d0 0x7f4ed40a3680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4005950 tx=0x7f4ee400b410 comp rx=0 tx=0).stop 2023-07-14T19:02:28.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80fee40 msgr2=0x7f4ee8198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80fee40 0x7f4ee8198110 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800b530 tx=0x7f4ed800b840 comp rx=0 tx=0).stop 2023-07-14T19:02:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 shutdown_connections 2023-07-14T19:02:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4ed40a11d0 0x7f4ed40a3680 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4005950 tx=0x7f4ee400b410 comp rx=0 tx=0).stop 2023-07-14T19:02:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4ee80fe4f0 0x7f4ee8197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ee80ff7f0 0x7f4ee819f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 --2- 172.21.15.78:0/692709351 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4ee80fee40 0x7f4ee8198110 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ed800b530 tx=0x7f4ed800b840 comp rx=0 tx=0).stop 2023-07-14T19:02:28.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.925+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 >> 172.21.15.78:0/692709351 conn(0x7f4ee80fa160 msgr2=0x7f4ee80fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.926+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 shutdown_connections 2023-07-14T19:02:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:28.926+0000 7f4eef3e8700 1 -- 172.21.15.78:0/692709351 wait complete. 2023-07-14T19:02:28.931 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' created 2023-07-14T19:02:28.942 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-07-14T19:02:29.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 -- 172.21.15.78:0/874931288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 msgr2=0x7f55b8101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:29.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 --2- 172.21.15.78:0/874931288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 0x7f55b8101270 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f55a4009990 tx=0x7f55a4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 -- 172.21.15.78:0/874931288 shutdown_connections 2023-07-14T19:02:29.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 --2- 172.21.15.78:0/874931288 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55b80ffc30 0x7f55b8100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 --2- 172.21.15.78:0/874931288 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b81017b0 0x7f55b8108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 --2- 172.21.15.78:0/874931288 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 0x7f55b8101270 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 -- 172.21.15.78:0/874931288 >> 172.21.15.78:0/874931288 conn(0x7f55b80fb290 msgr2=0x7f55b80fd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 -- 172.21.15.78:0/874931288 shutdown_connections 2023-07-14T19:02:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 -- 172.21.15.78:0/874931288 wait complete. 2023-07-14T19:02:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.070+0000 7f55bef18700 1 Processor -- start 2023-07-14T19:02:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bef18700 1 -- start start 2023-07-14T19:02:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bef18700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55b80ffc30 0x7f55b81958b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:29.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bef18700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 0x7f55b8195df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:29.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bef18700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b81017b0 0x7f55b819ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:29.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bef18700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819d430 con 0x7f55b8100e20 2023-07-14T19:02:29.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bef18700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819d570 con 0x7f55b80ffc30 2023-07-14T19:02:29.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bef18700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55b819d700 con 0x7f55b81017b0 2023-07-14T19:02:29.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 0x7f55b8195df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:29.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 0x7f55b8195df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34422/0 (socket says 172.21.15.78:34422) 2023-07-14T19:02:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 -- 172.21.15.78:0/2226140776 learned_addr learned my addr 172.21.15.78:0/2226140776 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bccb4700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55b80ffc30 0x7f55b81958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bd4b5700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b81017b0 0x7f55b819ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 -- 172.21.15.78:0/2226140776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b81017b0 msgr2=0x7f55b819ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b81017b0 0x7f55b819ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 -- 172.21.15.78:0/2226140776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55b80ffc30 msgr2=0x7f55b81958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55b80ffc30 0x7f55b81958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 -- 172.21.15.78:0/2226140776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a40095f0 con 0x7f55b8100e20 2023-07-14T19:02:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b7fff700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 0x7f55b8195df0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f55a4005f50 tx=0x7f55a40047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b5ffb700 1 -- 172.21.15.78:0/2226140776 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a401d070 con 0x7f55b8100e20 2023-07-14T19:02:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b5ffb700 1 -- 172.21.15.78:0/2226140776 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f55a400be50 con 0x7f55b8100e20 2023-07-14T19:02:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55b819d980 con 0x7f55b8100e20 2023-07-14T19:02:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.071+0000 7f55b5ffb700 1 -- 172.21.15.78:0/2226140776 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a4005320 con 0x7f55b8100e20 2023-07-14T19:02:29.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.072+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55b819de00 con 0x7f55b8100e20 2023-07-14T19:02:29.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.073+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55b810b2b0 con 0x7f55b8100e20 2023-07-14T19:02:29.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.074+0000 7f55b5ffb700 1 -- 172.21.15.78:0/2226140776 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f55a4004a80 con 0x7f55b8100e20 2023-07-14T19:02:29.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.074+0000 7f55b5ffb700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f55a80a11d0 0x7f55a80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:29.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.075+0000 7f55bccb4700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f55a80a11d0 0x7f55a80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:29.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.075+0000 7f55b5ffb700 1 -- 172.21.15.78:0/2226140776 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f55a40cdad0 con 0x7f55b8100e20 2023-07-14T19:02:29.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.075+0000 7f55bccb4700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f55a80a11d0 0x7f55a80a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f55b8100c80 tx=0x7f55ac009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:29.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.078+0000 7f55b5ffb700 1 -- 172.21.15.78:0/2226140776 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f55a40975e0 con 0x7f55b8100e20 2023-07-14T19:02:29.291 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.287+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f55b8106bd0 con 0x7f55b8100e20 2023-07-14T19:02:29.880 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:29 smithi102 ceph-mon[129280]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:29.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:29 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:02:29.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:29 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/692709351' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:02:29.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:29 smithi102 ceph-mon[129280]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:02:29.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.925+0000 7f55b5ffb700 1 -- 172.21.15.78:0/2226140776 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v79) v1 ==== 171+0+0 (secure 0 0 0) 0x7f55a409ac00 con 0x7f55b8100e20 2023-07-14T19:02:29.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.931+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f55a80a11d0 msgr2=0x7f55a80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:29.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.931+0000 7f55bef18700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f55a80a11d0 0x7f55a80a3680 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f55b8100c80 tx=0x7f55ac009260 comp rx=0 tx=0).stop 2023-07-14T19:02:29.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.931+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 msgr2=0x7f55b8195df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:29.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.931+0000 7f55bef18700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 0x7f55b8195df0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f55a4005f50 tx=0x7f55a40047a0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.938+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 shutdown_connections 2023-07-14T19:02:29.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.938+0000 7f55bef18700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f55a80a11d0 0x7f55a80a3680 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.938+0000 7f55bef18700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f55b80ffc30 0x7f55b81958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.938+0000 7f55bef18700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f55b81017b0 0x7f55b819ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.938+0000 7f55bef18700 1 --2- 172.21.15.78:0/2226140776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f55b8100e20 0x7f55b8195df0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:29.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.938+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 >> 172.21.15.78:0/2226140776 conn(0x7f55b80fb290 msgr2=0x7f55b80fe5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:29.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.941+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 shutdown_connections 2023-07-14T19:02:29.945 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:29.941+0000 7f55bef18700 1 -- 172.21.15.78:0/2226140776 wait complete. 2023-07-14T19:02:29.947 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' created 2023-07-14T19:02:29.957 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:29 smithi078 ceph-mon[127425]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:29.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:29 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:02:29.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:29 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/692709351' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:02:29.958 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:29 smithi078 ceph-mon[127425]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:02:29.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-07-14T19:02:29.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:29 smithi088 ceph-mon[128973]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:29.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:29 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:02:29.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:29 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/692709351' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2023-07-14T19:02:29.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:29 smithi088 ceph-mon[128973]: osdmap e78: 12 total, 12 up, 12 in 2023-07-14T19:02:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 -- 172.21.15.78:0/447270578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98200feb30 msgr2=0x7f98200fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 --2- 172.21.15.78:0/447270578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98200feb30 0x7f98200fef40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9810009990 tx=0x7f9810009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 -- 172.21.15.78:0/447270578 shutdown_connections 2023-07-14T19:02:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 --2- 172.21.15.78:0/447270578 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98201006b0 0x7f9820106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 --2- 172.21.15.78:0/447270578 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98200feb30 0x7f98200fef40 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 --2- 172.21.15.78:0/447270578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98200ffd20 0x7f9820100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 -- 172.21.15.78:0/447270578 >> 172.21.15.78:0/447270578 conn(0x7f98200fa150 msgr2=0x7f98200fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 -- 172.21.15.78:0/447270578 shutdown_connections 2023-07-14T19:02:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.085+0000 7f98256d7700 1 -- 172.21.15.78:0/447270578 wait complete. 2023-07-14T19:02:30.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f98256d7700 1 Processor -- start 2023-07-14T19:02:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f98256d7700 1 -- start start 2023-07-14T19:02:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f98256d7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98200feb30 0x7f9820069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f98256d7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200ffd20 0x7f982019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f98256d7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98201006b0 0x7f982019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f98256d7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982019d3e0 con 0x7f98201006b0 2023-07-14T19:02:30.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f98256d7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982019d550 con 0x7f98200ffd20 2023-07-14T19:02:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f98256d7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f982019d710 con 0x7f98200feb30 2023-07-14T19:02:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f981f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98201006b0 0x7f982019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.086+0000 7f981f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98201006b0 0x7f982019cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34446/0 (socket says 172.21.15.78:34446) 2023-07-14T19:02:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981f7fe700 1 -- 172.21.15.78:0/1426824234 learned_addr learned my addr 172.21.15.78:0/1426824234 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981effd700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98200feb30 0x7f9820069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981e7fc700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200ffd20 0x7f982019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981f7fe700 1 -- 172.21.15.78:0/1426824234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98200feb30 msgr2=0x7f9820069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:30.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981f7fe700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98200feb30 0x7f9820069560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981f7fe700 1 -- 172.21.15.78:0/1426824234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200ffd20 msgr2=0x7f982019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981f7fe700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200ffd20 0x7f982019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981f7fe700 1 -- 172.21.15.78:0/1426824234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100095f0 con 0x7f98201006b0 2023-07-14T19:02:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f981f7fe700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98201006b0 0x7f982019cc10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f981800ef30 tx=0x7f981800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f9807fff700 1 -- 172.21.15.78:0/1426824234 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818018070 con 0x7f98201006b0 2023-07-14T19:02:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f982019d990 con 0x7f98201006b0 2023-07-14T19:02:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f9807fff700 1 -- 172.21.15.78:0/1426824234 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f981800f040 con 0x7f98201006b0 2023-07-14T19:02:30.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f9807fff700 1 -- 172.21.15.78:0/1426824234 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981800cbc0 con 0x7f98201006b0 2023-07-14T19:02:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.087+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f982019dea0 con 0x7f98201006b0 2023-07-14T19:02:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.088+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9820105ab0 con 0x7f98201006b0 2023-07-14T19:02:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.089+0000 7f9807fff700 1 -- 172.21.15.78:0/1426824234 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f981800cd40 con 0x7f98201006b0 2023-07-14T19:02:30.096 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.090+0000 7f9807fff700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f98080a1150 0x7f98080a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.090+0000 7f981effd700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f98080a1150 0x7f98080a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.090+0000 7f9807fff700 1 -- 172.21.15.78:0/1426824234 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f98180cc7b0 con 0x7f98201006b0 2023-07-14T19:02:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.090+0000 7f981effd700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f98080a1150 0x7f98080a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9810006010 tx=0x7f981000b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:30.097 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.093+0000 7f9807fff700 1 -- 172.21.15.78:0/1426824234 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9818096140 con 0x7f98201006b0 2023-07-14T19:02:30.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.339+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f982002cb30 con 0x7f98201006b0 2023-07-14T19:02:30.880 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:30 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2226140776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T19:02:30.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:30 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2226140776' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T19:02:30.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:30 smithi102 ceph-mon[129280]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:02:30.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:30 smithi102 ceph-mon[129280]: pgmap v249: 97 pgs: 8 creating+peering, 1 active+clean, 88 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:30.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:30 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1426824234' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:02:30.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:30 smithi102 ceph-mon[129280]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:02:30.932 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.928+0000 7f9807fff700 1 -- 172.21.15.78:0/1426824234 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9818099760 con 0x7f98201006b0 2023-07-14T19:02:30.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f98080a1150 msgr2=0x7f98080a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f98080a1150 0x7f98080a3600 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9810006010 tx=0x7f981000b2a0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98201006b0 msgr2=0x7f982019cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98201006b0 0x7f982019cc10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f981800ef30 tx=0x7f981800c3a0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 shutdown_connections 2023-07-14T19:02:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f98080a1150 0x7f98080a3600 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f98201006b0 0x7f982019cc10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f98200feb30 0x7f9820069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 --2- 172.21.15.78:0/1426824234 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f98200ffd20 0x7f982019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.930+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 >> 172.21.15.78:0/1426824234 conn(0x7f98200fa150 msgr2=0x7f98200fc430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.931+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 shutdown_connections 2023-07-14T19:02:30.936 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:30.931+0000 7f98256d7700 1 -- 172.21.15.78:0/1426824234 wait complete. 2023-07-14T19:02:30.937 INFO:teuthology.orchestra.run.smithi078.stderr:new fs with metadata pool 2 and data pool 3 2023-07-14T19:02:30.947 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:30 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2226140776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T19:02:30.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:30 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2226140776' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T19:02:30.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:30 smithi078 ceph-mon[127425]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:02:30.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:30 smithi078 ceph-mon[127425]: pgmap v249: 97 pgs: 8 creating+peering, 1 active+clean, 88 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:30.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:30 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1426824234' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:02:30.948 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:30 smithi078 ceph-mon[127425]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:02:30.949 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-07-14T19:02:30.950 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-07-14T19:02:30.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:30 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2226140776' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2023-07-14T19:02:30.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:30 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2226140776' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2023-07-14T19:02:30.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:30 smithi088 ceph-mon[128973]: osdmap e79: 12 total, 12 up, 12 in 2023-07-14T19:02:30.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:30 smithi088 ceph-mon[128973]: pgmap v249: 97 pgs: 8 creating+peering, 1 active+clean, 88 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:02:30.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:30 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1426824234' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2023-07-14T19:02:30.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:30 smithi088 ceph-mon[128973]: daemon mds.b assigned to filesystem cephfs as rank 0 2023-07-14T19:02:31.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 -- 172.21.15.78:0/4157609315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8103010 msgr2=0x7fe4a81053f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:31.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 --2- 172.21.15.78:0/4157609315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8103010 0x7fe4a81053f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4009990 tx=0x7fe4a4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 -- 172.21.15.78:0/4157609315 shutdown_connections 2023-07-14T19:02:31.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 --2- 172.21.15.78:0/4157609315 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a81006f0 0x7fe4a8102ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 --2- 172.21.15.78:0/4157609315 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4a8105930 0x7fe4a8107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 --2- 172.21.15.78:0/4157609315 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8103010 0x7fe4a81053f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 -- 172.21.15.78:0/4157609315 >> 172.21.15.78:0/4157609315 conn(0x7fe4a80fa130 msgr2=0x7fe4a80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:31.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 -- 172.21.15.78:0/4157609315 shutdown_connections 2023-07-14T19:02:31.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.079+0000 7fe4af645700 1 -- 172.21.15.78:0/4157609315 wait complete. 2023-07-14T19:02:31.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4af645700 1 Processor -- start 2023-07-14T19:02:31.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4af645700 1 -- start start 2023-07-14T19:02:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4af645700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4a81006f0 0x7fe4a8195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4af645700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a8103010 0x7fe4a8195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4af645700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8105930 0x7fe4a819ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4af645700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a819d3d0 con 0x7fe4a8105930 2023-07-14T19:02:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4af645700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a819d510 con 0x7fe4a8103010 2023-07-14T19:02:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4af645700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a819d6a0 con 0x7fe4a81006f0 2023-07-14T19:02:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4adbe2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8105930 0x7fe4a819ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4adbe2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8105930 0x7fe4a819ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34460/0 (socket says 172.21.15.78:34460) 2023-07-14T19:02:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4adbe2700 1 -- 172.21.15.78:0/1620986006 learned_addr learned my addr 172.21.15.78:0/1620986006 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4acbe0700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a8103010 0x7fe4a8195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.080+0000 7fe4ad3e1700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4a81006f0 0x7fe4a8195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe4adbe2700 1 -- 172.21.15.78:0/1620986006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4a81006f0 msgr2=0x7fe4a8195850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe4adbe2700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4a81006f0 0x7fe4a8195850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe4adbe2700 1 -- 172.21.15.78:0/1620986006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a8103010 msgr2=0x7fe4a8195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe4adbe2700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a8103010 0x7fe4a8195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe4adbe2700 1 -- 172.21.15.78:0/1620986006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40095f0 con 0x7fe4a8105930 2023-07-14T19:02:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe4adbe2700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8105930 0x7fe4a819ce00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0007ee0 tx=0x7fe4a000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe49e7fc700 1 -- 172.21.15.78:0/1620986006 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0017070 con 0x7fe4a8105930 2023-07-14T19:02:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a819d980 con 0x7fe4a8105930 2023-07-14T19:02:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe49e7fc700 1 -- 172.21.15.78:0/1620986006 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe4a000f040 con 0x7fe4a8105930 2023-07-14T19:02:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe49e7fc700 1 -- 172.21.15.78:0/1620986006 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a0005710 con 0x7fe4a8105930 2023-07-14T19:02:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.081+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a819de40 con 0x7fe4a8105930 2023-07-14T19:02:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.082+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a80fbca0 con 0x7fe4a8105930 2023-07-14T19:02:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.083+0000 7fe49e7fc700 1 -- 172.21.15.78:0/1620986006 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe4a0005870 con 0x7fe4a8105930 2023-07-14T19:02:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.083+0000 7fe49e7fc700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1150 0x7fe4940a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.083+0000 7fe4ad3e1700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1150 0x7fe4940a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.083+0000 7fe49e7fc700 1 -- 172.21.15.78:0/1620986006 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe4a00cbfb0 con 0x7fe4a8105930 2023-07-14T19:02:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.084+0000 7fe4ad3e1700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1150 0x7fe4940a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe49800bbd0 tx=0x7fe49800b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.088+0000 7fe49e7fc700 1 -- 172.21.15.78:0/1620986006 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe4a00fe780 con 0x7fe4a8105930 2023-07-14T19:02:31.309 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.306+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7fe4a802cb10 con 0x7fe4a8105930 2023-07-14T19:02:31.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.554+0000 7fe49e7fc700 1 -- 172.21.15.78:0/1620986006 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v81) v1 ==== 157+0+0 (secure 0 0 0) 0x7fe4a0095850 con 0x7fe4a8105930 2023-07-14T19:02:31.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.556+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1150 msgr2=0x7fe4940a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.556+0000 7fe4af645700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1150 0x7fe4940a3600 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe49800bbd0 tx=0x7fe49800b5b0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.556+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8105930 msgr2=0x7fe4a819ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.556+0000 7fe4af645700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8105930 0x7fe4a819ce00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0007ee0 tx=0x7fe4a000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.557+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 shutdown_connections 2023-07-14T19:02:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.557+0000 7fe4af645700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1150 0x7fe4940a3600 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.557+0000 7fe4af645700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4a8105930 0x7fe4a819ce00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.557+0000 7fe4af645700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4a81006f0 0x7fe4a8195850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.557+0000 7fe4af645700 1 --2- 172.21.15.78:0/1620986006 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4a8103010 0x7fe4a8195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.557+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 >> 172.21.15.78:0/1620986006 conn(0x7fe4a80fa130 msgr2=0x7fe4a8101350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.557+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 shutdown_connections 2023-07-14T19:02:31.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.557+0000 7fe4af645700 1 -- 172.21.15.78:0/1620986006 wait complete. 2023-07-14T19:02:31.573 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2023-07-14T19:02:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 -- 172.21.15.78:0/1462953658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0100720 msgr2=0x7fcac0102b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 --2- 172.21.15.78:0/1462953658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0100720 0x7fcac0102b00 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcabc009990 tx=0x7fcabc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 -- 172.21.15.78:0/1462953658 shutdown_connections 2023-07-14T19:02:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 --2- 172.21.15.78:0/1462953658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcac0105960 0x7fcac0107d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 --2- 172.21.15.78:0/1462953658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0100720 0x7fcac0102b00 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 --2- 172.21.15.78:0/1462953658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcac0103040 0x7fcac0105420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 -- 172.21.15.78:0/1462953658 >> 172.21.15.78:0/1462953658 conn(0x7fcac00fa120 msgr2=0x7fcac00fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 -- 172.21.15.78:0/1462953658 shutdown_connections 2023-07-14T19:02:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 -- 172.21.15.78:0/1462953658 wait complete. 2023-07-14T19:02:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 Processor -- start 2023-07-14T19:02:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 -- start start 2023-07-14T19:02:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcac0100720 0x7fcac0100090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.702+0000 7fcac80b8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcac0103040 0x7fcac00fe6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac80b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0105960 0x7fcac00fec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac80b8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00ff3b0 con 0x7fcac0105960 2023-07-14T19:02:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac80b8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac00ff520 con 0x7fcac0100720 2023-07-14T19:02:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac80b8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac019f820 con 0x7fcac0103040 2023-07-14T19:02:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0105960 0x7fcac00fec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0105960 0x7fcac00fec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34474/0 (socket says 172.21.15.78:34474) 2023-07-14T19:02:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 -- 172.21.15.78:0/3697030465 learned_addr learned my addr 172.21.15.78:0/3697030465 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac5e54700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcac0100720 0x7fcac0100090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac5653700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcac0103040 0x7fcac00fe6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 -- 172.21.15.78:0/3697030465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcac0103040 msgr2=0x7fcac00fe6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcac0103040 0x7fcac00fe6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 -- 172.21.15.78:0/3697030465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcac0100720 msgr2=0x7fcac0100090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcac0100720 0x7fcac0100090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 -- 172.21.15.78:0/3697030465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcabc0095f0 con 0x7fcac0105960 2023-07-14T19:02:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac6655700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0105960 0x7fcac00fec10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcab800ef30 tx=0x7fcab800c450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcab6ffd700 1 -- 172.21.15.78:0/3697030465 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab8018070 con 0x7fcac0105960 2023-07-14T19:02:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcab6ffd700 1 -- 172.21.15.78:0/3697030465 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcab800f040 con 0x7fcac0105960 2023-07-14T19:02:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac019faa0 con 0x7fcac0105960 2023-07-14T19:02:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcab6ffd700 1 -- 172.21.15.78:0/3697030465 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcab8003d00 con 0x7fcac0105960 2023-07-14T19:02:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.703+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac019ffb0 con 0x7fcac0105960 2023-07-14T19:02:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.704+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac00fbc90 con 0x7fcac0105960 2023-07-14T19:02:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.705+0000 7fcab6ffd700 1 -- 172.21.15.78:0/3697030465 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fcab800caa0 con 0x7fcac0105960 2023-07-14T19:02:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.706+0000 7fcab6ffd700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcaac0a1270 0x7fcaac0a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.706+0000 7fcab6ffd700 1 -- 172.21.15.78:0/3697030465 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8205+0+0 (secure 0 0 0) 0x7fcab80ccf60 con 0x7fcac0105960 2023-07-14T19:02:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.706+0000 7fcac5e54700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcaac0a1270 0x7fcaac0a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.706+0000 7fcac5e54700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcaac0a1270 0x7fcaac0a3720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcabc006010 tx=0x7fcabc00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.709+0000 7fcab6ffd700 1 -- 172.21.15.78:0/3697030465 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcab8096760 con 0x7fcac0105960 2023-07-14T19:02:31.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:02:31.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1426824234' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:02:31.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:02:31.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:02:31.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:31.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:02:31.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.883 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:31 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1620986006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T19:02:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:31.898+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fcac002c8e0 con 0x7fcac0105960 2023-07-14T19:02:31.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:02:31.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1426824234' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:02:31.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:02:31.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:02:31.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:31.963 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:02:31.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:31.964 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:31 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1620986006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T19:02:32.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: osdmap e80: 12 total, 12 up, 12 in 2023-07-14T19:02:32.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1426824234' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2023-07-14T19:02:32.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2023-07-14T19:02:32.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: daemon mds.b is now active in filesystem cephfs as rank 0 2023-07-14T19:02:32.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:32.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:32.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:32.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:32.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:32.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:32.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:02:32.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:02:32.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:02:32.049 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:31 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1620986006' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2023-07-14T19:02:33.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:32 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1620986006' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T19:02:33.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:32 smithi102 ceph-mon[129280]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:02:33.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:32 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3697030465' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:02:33.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:32 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] up:active 2023-07-14T19:02:33.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:32 smithi102 ceph-mon[129280]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:33.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:32 smithi102 ceph-mon[129280]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:33.132 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:32 smithi102 ceph-mon[129280]: pgmap v252: 97 pgs: 52 active+clean, 8 creating+peering, 37 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2023-07-14T19:02:33.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:32 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1620986006' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T19:02:33.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:32 smithi088 ceph-mon[128973]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:02:33.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:32 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3697030465' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:02:33.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:32 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] up:active 2023-07-14T19:02:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:32 smithi088 ceph-mon[128973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:32 smithi088 ceph-mon[128973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:33.213 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:32 smithi088 ceph-mon[128973]: pgmap v252: 97 pgs: 52 active+clean, 8 creating+peering, 37 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2023-07-14T19:02:33.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:32 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1620986006' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2023-07-14T19:02:33.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:32 smithi078 ceph-mon[127425]: osdmap e81: 12 total, 12 up, 12 in 2023-07-14T19:02:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:32 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3697030465' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2023-07-14T19:02:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:32 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] up:active 2023-07-14T19:02:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:32 smithi078 ceph-mon[127425]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:32 smithi078 ceph-mon[127425]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:33.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:32 smithi078 ceph-mon[127425]: pgmap v252: 97 pgs: 52 active+clean, 8 creating+peering, 37 unknown; 577 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2023-07-14T19:02:33.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.563+0000 7fcab6ffd700 1 -- 172.21.15.78:0/3697030465 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v83) v1 ==== 228+0+0 (secure 0 0 0) 0x7fcab8099d80 con 0x7fcac0105960 2023-07-14T19:02:33.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcaac0a1270 msgr2=0x7fcaac0a3720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:33.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcaac0a1270 0x7fcaac0a3720 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcabc006010 tx=0x7fcabc00b260 comp rx=0 tx=0).stop 2023-07-14T19:02:33.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0105960 msgr2=0x7fcac00fec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:33.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0105960 0x7fcac00fec10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcab800ef30 tx=0x7fcab800c450 comp rx=0 tx=0).stop 2023-07-14T19:02:33.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 shutdown_connections 2023-07-14T19:02:33.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcaac0a1270 0x7fcaac0a3720 secure :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcabc006010 tx=0x7fcabc00b260 comp rx=0 tx=0).stop 2023-07-14T19:02:33.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcac0100720 0x7fcac0100090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcac0105960 0x7fcac00fec10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.571+0000 7fcac80b8700 1 --2- 172.21.15.78:0/3697030465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcac0103040 0x7fcac00fe6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.572+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 >> 172.21.15.78:0/3697030465 conn(0x7fcac00fa120 msgr2=0x7fcac0101440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:33.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.572+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 shutdown_connections 2023-07-14T19:02:33.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.572+0000 7fcac80b8700 1 -- 172.21.15.78:0/3697030465 wait complete. 2023-07-14T19:02:33.578 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' created 2023-07-14T19:02:33.590 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2023-07-14T19:02:33.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.717+0000 7f59cee3e700 1 -- 172.21.15.78:0/2946629372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c8102fb0 msgr2=0x7f59c8105390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:33.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.717+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2946629372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c8102fb0 0x7f59c8105390 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f59b4009990 tx=0x7f59b4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.717+0000 7f59cee3e700 1 -- 172.21.15.78:0/2946629372 shutdown_connections 2023-07-14T19:02:33.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.717+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2946629372 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c81058d0 0x7f59c8107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.717+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2946629372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59c8100690 0x7f59c8102a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.717+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2946629372 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c8102fb0 0x7f59c8105390 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 -- 172.21.15.78:0/2946629372 >> 172.21.15.78:0/2946629372 conn(0x7f59c80fa130 msgr2=0x7f59c80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:33.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 -- 172.21.15.78:0/2946629372 shutdown_connections 2023-07-14T19:02:33.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 -- 172.21.15.78:0/2946629372 wait complete. 2023-07-14T19:02:33.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 Processor -- start 2023-07-14T19:02:33.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 -- start start 2023-07-14T19:02:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c8100690 0x7f59c8100000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59c8102fb0 0x7f59c80fe640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c81058d0 0x7f59c80feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80ff320 con 0x7f59c81058d0 2023-07-14T19:02:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c80ff490 con 0x7f59c8100690 2023-07-14T19:02:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cee3e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c819f7c0 con 0x7f59c8102fb0 2023-07-14T19:02:33.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cd3db700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c81058d0 0x7f59c80feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cd3db700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c81058d0 0x7f59c80feb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34488/0 (socket says 172.21.15.78:34488) 2023-07-14T19:02:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.718+0000 7f59cd3db700 1 -- 172.21.15.78:0/2150472245 learned_addr learned my addr 172.21.15.78:0/2150472245 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59c7fff700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59c8102fb0 0x7f59c80fe640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59ccbda700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c8100690 0x7f59c8100000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59cd3db700 1 -- 172.21.15.78:0/2150472245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59c8102fb0 msgr2=0x7f59c80fe640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59cd3db700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59c8102fb0 0x7f59c80fe640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.726 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59cd3db700 1 -- 172.21.15.78:0/2150472245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c8100690 msgr2=0x7f59c8100000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59cd3db700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c8100690 0x7f59c8100000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59cd3db700 1 -- 172.21.15.78:0/2150472245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b40095f0 con 0x7f59c81058d0 2023-07-14T19:02:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59cd3db700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c81058d0 0x7f59c80feb80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f59c0007d70 tx=0x7f59c000a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59c5ffb700 1 -- 172.21.15.78:0/2150472245 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c000ad40 con 0x7f59c81058d0 2023-07-14T19:02:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c819faa0 con 0x7f59c81058d0 2023-07-14T19:02:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59c5ffb700 1 -- 172.21.15.78:0/2150472245 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f59c00128b0 con 0x7f59c81058d0 2023-07-14T19:02:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59c5ffb700 1 -- 172.21.15.78:0/2150472245 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c00115c0 con 0x7f59c81058d0 2023-07-14T19:02:33.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.719+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c81a0000 con 0x7f59c81058d0 2023-07-14T19:02:33.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.720+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c80fbca0 con 0x7f59c81058d0 2023-07-14T19:02:33.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.721+0000 7f59c5ffb700 1 -- 172.21.15.78:0/2150472245 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f59c0012a20 con 0x7f59c81058d0 2023-07-14T19:02:33.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.722+0000 7f59c5ffb700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f59b80a1150 0x7f59b80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.722+0000 7f59ccbda700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f59b80a1150 0x7f59b80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.722+0000 7f59c5ffb700 1 -- 172.21.15.78:0/2150472245 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f59c00cd120 con 0x7f59c81058d0 2023-07-14T19:02:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.722+0000 7f59ccbda700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f59b80a1150 0x7f59b80a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f59bc005ea0 tx=0x7f59bc005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:33.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.725+0000 7f59c5ffb700 1 -- 172.21.15.78:0/2150472245 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f59c00966a0 con 0x7f59c81058d0 2023-07-14T19:02:33.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:33.914+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f59c802cd70 con 0x7f59c81058d0 2023-07-14T19:02:34.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:33 smithi078 ceph-mon[127425]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:02:34.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:33 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3697030465' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:02:34.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:33 smithi078 ceph-mon[127425]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:02:34.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:33 smithi102 ceph-mon[129280]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:02:34.162 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:33 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3697030465' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:02:34.163 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:33 smithi102 ceph-mon[129280]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:02:34.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:33 smithi088 ceph-mon[128973]: osdmap e82: 12 total, 12 up, 12 in 2023-07-14T19:02:34.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:33 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3697030465' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2023-07-14T19:02:34.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:33 smithi088 ceph-mon[128973]: osdmap e83: 12 total, 12 up, 12 in 2023-07-14T19:02:34.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.565+0000 7f59c5ffb700 1 -- 172.21.15.78:0/2150472245 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v84) v1 ==== 169+0+0 (secure 0 0 0) 0x7f59c0023350 con 0x7f59c81058d0 2023-07-14T19:02:34.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.572+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f59b80a1150 msgr2=0x7f59b80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:34.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.572+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f59b80a1150 0x7f59b80a3600 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f59bc005ea0 tx=0x7f59bc005e30 comp rx=0 tx=0).stop 2023-07-14T19:02:34.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.572+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c81058d0 msgr2=0x7f59c80feb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:34.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.572+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c81058d0 0x7f59c80feb80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f59c0007d70 tx=0x7f59c000a340 comp rx=0 tx=0).stop 2023-07-14T19:02:34.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.574+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 shutdown_connections 2023-07-14T19:02:34.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.574+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f59b80a1150 0x7f59b80a3600 secure :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f59bc005ea0 tx=0x7f59bc005e30 comp rx=0 tx=0).stop 2023-07-14T19:02:34.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.574+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f59c8100690 0x7f59c8100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.574+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f59c81058d0 0x7f59c80feb80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.574+0000 7f59cee3e700 1 --2- 172.21.15.78:0/2150472245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f59c8102fb0 0x7f59c80fe640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.574+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 >> 172.21.15.78:0/2150472245 conn(0x7f59c80fa130 msgr2=0x7f59c81012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:34.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.580+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 shutdown_connections 2023-07-14T19:02:34.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.580+0000 7f59cee3e700 1 -- 172.21.15.78:0/2150472245 wait complete. 2023-07-14T19:02:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:set pool 4 allow_ec_overwrites to true 2023-07-14T19:02:34.604 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2023-07-14T19:02:34.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 -- 172.21.15.78:0/4240945241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 msgr2=0x7f5038100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:34.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 --2- 172.21.15.78:0/4240945241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 0x7f5038100150 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f502c009990 tx=0x7f502c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 -- 172.21.15.78:0/4240945241 shutdown_connections 2023-07-14T19:02:34.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 --2- 172.21.15.78:0/4240945241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f50380feb10 0x7f50380fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 --2- 172.21.15.78:0/4240945241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5038100690 0x7f5038106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 --2- 172.21.15.78:0/4240945241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 0x7f5038100150 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 -- 172.21.15.78:0/4240945241 >> 172.21.15.78:0/4240945241 conn(0x7f50380fa130 msgr2=0x7f50380fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:34.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 -- 172.21.15.78:0/4240945241 shutdown_connections 2023-07-14T19:02:34.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.727+0000 7f503cd17700 1 -- 172.21.15.78:0/4240945241 wait complete. 2023-07-14T19:02:34.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f503cd17700 1 Processor -- start 2023-07-14T19:02:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f503cd17700 1 -- start start 2023-07-14T19:02:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f503cd17700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50380feb10 0x7f5038068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f503cd17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 0x7f503819a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f503cd17700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5038100690 0x7f503819ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f503cd17700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503819d450 con 0x7f50380ffd00 2023-07-14T19:02:34.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f503cd17700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503819d5c0 con 0x7f5038100690 2023-07-14T19:02:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f503cd17700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f503819d780 con 0x7f50380feb10 2023-07-14T19:02:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.728+0000 7f5035d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 0x7f503819a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5035d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 0x7f503819a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34512/0 (socket says 172.21.15.78:34512) 2023-07-14T19:02:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5035d9b700 1 -- 172.21.15.78:0/461443891 learned_addr learned my addr 172.21.15.78:0/461443891 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5036d9d700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5038100690 0x7f503819ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5035d9b700 1 -- 172.21.15.78:0/461443891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50380feb10 msgr2=0x7f5038068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f503659c700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50380feb10 0x7f5038068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:34.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5035d9b700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50380feb10 0x7f5038068310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5035d9b700 1 -- 172.21.15.78:0/461443891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5038100690 msgr2=0x7f503819ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5035d9b700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5038100690 0x7f503819ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5035d9b700 1 -- 172.21.15.78:0/461443891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f502c0095f0 con 0x7f50380ffd00 2023-07-14T19:02:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5036d9d700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5038100690 0x7f503819ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f5035d9b700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 0x7f503819a670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f502c003a50 tx=0x7f502c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f50277fe700 1 -- 172.21.15.78:0/461443891 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c01d070 con 0x7f50380ffd00 2023-07-14T19:02:34.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f50277fe700 1 -- 172.21.15.78:0/461443891 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f502c005a00 con 0x7f50380ffd00 2023-07-14T19:02:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f503819da00 con 0x7f50380ffd00 2023-07-14T19:02:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f50277fe700 1 -- 172.21.15.78:0/461443891 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502c005280 con 0x7f50380ffd00 2023-07-14T19:02:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.729+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f503819ded0 con 0x7f50380ffd00 2023-07-14T19:02:34.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.730+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f503810a210 con 0x7f50380ffd00 2023-07-14T19:02:34.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.731+0000 7f50277fe700 1 -- 172.21.15.78:0/461443891 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f502c0053e0 con 0x7f50380ffd00 2023-07-14T19:02:34.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.732+0000 7f50277fe700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50200a1150 0x7f50200a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:34.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.732+0000 7f503659c700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50200a1150 0x7f50200a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:34.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.732+0000 7f50277fe700 1 -- 172.21.15.78:0/461443891 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 8698+0+0 (secure 0 0 0) 0x7f502c0cdc50 con 0x7f50380ffd00 2023-07-14T19:02:34.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.732+0000 7f503659c700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50200a1150 0x7f50200a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5028005950 tx=0x7f50280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:34.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.736+0000 7f50277fe700 1 -- 172.21.15.78:0/461443891 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f502c05b780 con 0x7f50380ffd00 2023-07-14T19:02:34.978 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:34.975+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f5038105a90 con 0x7f50380ffd00 2023-07-14T19:02:35.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:34 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2150472245' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T19:02:35.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:34 smithi078 ceph-mon[127425]: pgmap v255: 129 pgs: 71 active+clean, 9 creating+peering, 49 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2023-07-14T19:02:35.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:34 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2150472245' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T19:02:35.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:34 smithi078 ceph-mon[127425]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:02:35.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:34 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2150472245' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T19:02:35.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:34 smithi102 ceph-mon[129280]: pgmap v255: 129 pgs: 71 active+clean, 9 creating+peering, 49 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2023-07-14T19:02:35.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:34 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2150472245' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T19:02:35.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:34 smithi102 ceph-mon[129280]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:02:35.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:34 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2150472245' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2023-07-14T19:02:35.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:34 smithi088 ceph-mon[128973]: pgmap v255: 129 pgs: 71 active+clean, 9 creating+peering, 49 unknown; 579 KiB data, 328 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s wr, 4 op/s 2023-07-14T19:02:35.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:34 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2150472245' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2023-07-14T19:02:35.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:34 smithi088 ceph-mon[128973]: osdmap e84: 12 total, 12 up, 12 in 2023-07-14T19:02:35.712 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:02:35 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b[154122]: 2023-07-14T19:02:35.402+0000 7fa961a23700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-07-14T19:02:35.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.819+0000 7f50277fe700 1 -- 172.21.15.78:0/461443891 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) v1 ==== 137+0+0 (secure 0 0 0) 0x7f502c0d3020 con 0x7f50380ffd00 2023-07-14T19:02:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50200a1150 msgr2=0x7f50200a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50200a1150 0x7f50200a3600 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f5028005950 tx=0x7f50280058e0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 msgr2=0x7f503819a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:35.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 0x7f503819a670 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f502c003a50 tx=0x7f502c009580 comp rx=0 tx=0).stop 2023-07-14T19:02:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 shutdown_connections 2023-07-14T19:02:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50200a1150 0x7f50200a3600 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5038100690 0x7f503819ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50380feb10 0x7f5038068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 --2- 172.21.15.78:0/461443891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f50380ffd00 0x7f503819a670 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 >> 172.21.15.78:0/461443891 conn(0x7f50380fa130 msgr2=0x7f50380fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 shutdown_connections 2023-07-14T19:02:35.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.822+0000 7f503cd17700 1 -- 172.21.15.78:0/461443891 wait complete. 2023-07-14T19:02:35.828 INFO:teuthology.orchestra.run.smithi078.stderr:added data pool 4 to fsmap 2023-07-14T19:02:35.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:35.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 -- 172.21.15.78:0/815711674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 msgr2=0x7f9c600fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:35.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 --2- 172.21.15.78:0/815711674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 0x7f9c600fef10 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c009990 tx=0x7f9c5c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 -- 172.21.15.78:0/815711674 shutdown_connections 2023-07-14T19:02:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 --2- 172.21.15.78:0/815711674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c60100680 0x7f9c60106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 --2- 172.21.15.78:0/815711674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 0x7f9c600fef10 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 --2- 172.21.15.78:0/815711674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c600ffcf0 0x7f9c60100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 -- 172.21.15.78:0/815711674 >> 172.21.15.78:0/815711674 conn(0x7f9c600fa100 msgr2=0x7f9c600fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 -- 172.21.15.78:0/815711674 shutdown_connections 2023-07-14T19:02:35.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 -- 172.21.15.78:0/815711674 wait complete. 2023-07-14T19:02:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 Processor -- start 2023-07-14T19:02:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 -- start start 2023-07-14T19:02:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.963+0000 7f9c6802c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 0x7f9c60194710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c6802c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c600ffcf0 0x7f9c60194c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c6802c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c60100680 0x7f9c6019bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c6802c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6019c290 con 0x7f9c600feb00 2023-07-14T19:02:35.970 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c6802c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6019c3d0 con 0x7f9c600ffcf0 2023-07-14T19:02:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c6802c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6019c560 con 0x7f9c60100680 2023-07-14T19:02:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 0x7f9c60194710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 0x7f9c60194710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34524/0 (socket says 172.21.15.78:34524) 2023-07-14T19:02:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 -- 172.21.15.78:0/1444280253 learned_addr learned my addr 172.21.15.78:0/1444280253 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 -- 172.21.15.78:0/1444280253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c60100680 msgr2=0x7f9c6019bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c665c9700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c60100680 0x7f9c6019bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c60100680 0x7f9c6019bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 -- 172.21.15.78:0/1444280253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c600ffcf0 msgr2=0x7f9c60194c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c600ffcf0 0x7f9c60194c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 -- 172.21.15.78:0/1444280253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5c0095f0 con 0x7f9c600feb00 2023-07-14T19:02:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c65dc8700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 0x7f9c60194710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c004730 tx=0x7f9c5c009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c52ffd700 1 -- 172.21.15.78:0/1444280253 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c01f070 con 0x7f9c600feb00 2023-07-14T19:02:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c52ffd700 1 -- 172.21.15.78:0/1444280253 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9c5c004ca0 con 0x7f9c600feb00 2023-07-14T19:02:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c600677a0 con 0x7f9c600feb00 2023-07-14T19:02:35.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c52ffd700 1 -- 172.21.15.78:0/1444280253 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c5c005030 con 0x7f9c600feb00 2023-07-14T19:02:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.964+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c60067bf0 con 0x7f9c600feb00 2023-07-14T19:02:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.965+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c60068640 con 0x7f9c600feb00 2023-07-14T19:02:35.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.966+0000 7f9c52ffd700 1 -- 172.21.15.78:0/1444280253 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f9c5c023070 con 0x7f9c600feb00 2023-07-14T19:02:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.967+0000 7f9c52ffd700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9c4c0a11a0 0x7f9c4c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.967+0000 7f9c655c7700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9c4c0a11a0 0x7f9c4c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.967+0000 7f9c52ffd700 1 -- 172.21.15.78:0/1444280253 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f9c5c0d04e0 con 0x7f9c600feb00 2023-07-14T19:02:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.967+0000 7f9c655c7700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9c4c0a11a0 0x7f9c4c0a3650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c54005fd0 tx=0x7f9c54005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:35.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:35.970+0000 7f9c52ffd700 1 -- 172.21.15.78:0/1444280253 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9c5c099ad0 con 0x7f9c600feb00 2023-07-14T19:02:36.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:35 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/461443891' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T19:02:36.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:35 smithi102 ceph-mon[129280]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:02:36.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.205+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c60105d10 con 0x7f9c600feb00 2023-07-14T19:02:36.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.206+0000 7f9c52ffd700 1 -- 172.21.15.78:0/1444280253 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10748 (secure 0 0 0) 0x7f9c5c09d0f0 con 0x7f9c600feb00 2023-07-14T19:02:36.210 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:36.210 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:02:35.807625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-14T19:02:36.211 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:35 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/461443891' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T19:02:36.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:35 smithi088 ceph-mon[128973]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:02:36.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9c4c0a11a0 msgr2=0x7f9c4c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9c4c0a11a0 0x7f9c4c0a3650 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9c54005fd0 tx=0x7f9c54005d00 comp rx=0 tx=0).stop 2023-07-14T19:02:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 msgr2=0x7f9c60194710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 0x7f9c60194710 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f9c5c004730 tx=0x7f9c5c009580 comp rx=0 tx=0).stop 2023-07-14T19:02:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 shutdown_connections 2023-07-14T19:02:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9c4c0a11a0 0x7f9c4c0a3650 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c60100680 0x7f9c6019bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9c600feb00 0x7f9c60194710 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 --2- 172.21.15.78:0/1444280253 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9c600ffcf0 0x7f9c60194c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 >> 172.21.15.78:0/1444280253 conn(0x7f9c600fa100 msgr2=0x7f9c601031e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 shutdown_connections 2023-07-14T19:02:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.208+0000 7f9c6802c700 1 -- 172.21.15.78:0/1444280253 wait complete. 2023-07-14T19:02:36.214 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-07-14T19:02:36.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:35 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/461443891' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2023-07-14T19:02:36.223 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:35 smithi078 ceph-mon[127425]: osdmap e85: 12 total, 12 up, 12 in 2023-07-14T19:02:36.224 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:36.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 -- 172.21.15.78:0/2953195986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 msgr2=0x7f7b0c100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2953195986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 0x7f7b0c100140 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f7afc009990 tx=0x7f7afc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 -- 172.21.15.78:0/2953195986 shutdown_connections 2023-07-14T19:02:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2953195986 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b0c0feb00 0x7f7b0c0fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2953195986 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b0c100680 0x7f7b0c106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2953195986 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 0x7f7b0c100140 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 -- 172.21.15.78:0/2953195986 >> 172.21.15.78:0/2953195986 conn(0x7f7b0c0fa100 msgr2=0x7f7b0c0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 -- 172.21.15.78:0/2953195986 shutdown_connections 2023-07-14T19:02:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 -- 172.21.15.78:0/2953195986 wait complete. 2023-07-14T19:02:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 Processor -- start 2023-07-14T19:02:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.342+0000 7f7b139ae700 1 -- start start 2023-07-14T19:02:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b139ae700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b0c0feb00 0x7f7b0c068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:36.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b139ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 0x7f7b0c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:36.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b139ae700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b0c100680 0x7f7b0c19cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:36.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b139ae700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c19d390 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b139ae700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c19d500 con 0x7f7b0c100680 2023-07-14T19:02:36.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b139ae700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b0c19d6c0 con 0x7f7b0c0feb00 2023-07-14T19:02:36.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 0x7f7b0c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:36.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 0x7f7b0c19a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34540/0 (socket says 172.21.15.78:34540) 2023-07-14T19:02:36.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 -- 172.21.15.78:0/2089451994 learned_addr learned my addr 172.21.15.78:0/2089451994 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:36.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b11f4b700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b0c100680 0x7f7b0c19cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:36.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b1174a700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b0c0feb00 0x7f7b0c068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:36.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 -- 172.21.15.78:0/2089451994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b0c0feb00 msgr2=0x7f7b0c068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b0c0feb00 0x7f7b0c068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 -- 172.21.15.78:0/2089451994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b0c100680 msgr2=0x7f7b0c19cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b0c100680 0x7f7b0c19cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 -- 172.21.15.78:0/2089451994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7afc0095f0 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b1174a700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b0c0feb00 0x7f7b0c068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:36.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b10f49700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 0x7f7b0c19a5b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7afc003a50 tx=0x7f7afc009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:36.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b027fc700 1 -- 172.21.15.78:0/2089451994 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc01d070 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b027fc700 1 -- 172.21.15.78:0/2089451994 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7afc005a00 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b0c19d940 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b027fc700 1 -- 172.21.15.78:0/2089451994 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7afc005020 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.343+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b0c19de10 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.344+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b0c10a270 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.345+0000 7f7b027fc700 1 -- 172.21.15.78:0/2089451994 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7afc0051a0 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.357 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.346+0000 7f7b027fc700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7af80a11a0 0x7f7af80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:36.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.346+0000 7f7b1174a700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7af80a11a0 0x7f7af80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:36.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.346+0000 7f7b027fc700 1 -- 172.21.15.78:0/2089451994 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7afc0ce030 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.346+0000 7f7b1174a700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7af80a11a0 0x7f7af80a3650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b08005fd0 tx=0x7f7b08005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:36.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.349+0000 7f7b027fc700 1 -- 172.21.15.78:0/2089451994 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7afc097620 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.539+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b0c105a30 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.541+0000 7f7b027fc700 1 -- 172.21.15.78:0/2089451994 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7f7afc09ac40 con 0x7f7b0c0ffcf0 2023-07-14T19:02:36.545 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:36.545 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7af80a11a0 msgr2=0x7f7af80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7af80a11a0 0x7f7af80a3650 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f7b08005fd0 tx=0x7f7b08005f00 comp rx=0 tx=0).stop 2023-07-14T19:02:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 msgr2=0x7f7b0c19a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 0x7f7b0c19a5b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7afc003a50 tx=0x7f7afc009580 comp rx=0 tx=0).stop 2023-07-14T19:02:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 shutdown_connections 2023-07-14T19:02:36.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7af80a11a0 0x7f7af80a3650 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7b0c100680 0x7f7b0c19cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b0c0feb00 0x7f7b0c068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 --2- 172.21.15.78:0/2089451994 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7b0c0ffcf0 0x7f7b0c19a5b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 >> 172.21.15.78:0/2089451994 conn(0x7f7b0c0fa100 msgr2=0x7f7b0c0fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 shutdown_connections 2023-07-14T19:02:36.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.543+0000 7f7b139ae700 1 -- 172.21.15.78:0/2089451994 wait complete. 2023-07-14T19:02:36.559 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:36.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.675+0000 7f5ec84d8700 1 -- 172.21.15.78:0/3062854856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 msgr2=0x7f5ec0106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.675+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/3062854856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 0x7f5ec0106f70 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800b0a0 tx=0x7f5eb800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 -- 172.21.15.78:0/3062854856 shutdown_connections 2023-07-14T19:02:36.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/3062854856 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 0x7f5ec0106f70 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/3062854856 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec00feb30 0x7f5ec00fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/3062854856 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ec00ffd20 0x7f5ec0100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 -- 172.21.15.78:0/3062854856 >> 172.21.15.78:0/3062854856 conn(0x7f5ec00fa150 msgr2=0x7f5ec00fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:36.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 -- 172.21.15.78:0/3062854856 shutdown_connections 2023-07-14T19:02:36.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 -- 172.21.15.78:0/3062854856 wait complete. 2023-07-14T19:02:36.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 Processor -- start 2023-07-14T19:02:36.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 -- start start 2023-07-14T19:02:36.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec00feb30 0x7f5ec0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:36.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ec00ffd20 0x7f5ec0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:36.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 0x7f5ec019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:36.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec019f6b0 con 0x7f5ec01006b0 2023-07-14T19:02:36.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec019f7f0 con 0x7f5ec00ffd20 2023-07-14T19:02:36.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec84d8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec019f980 con 0x7f5ec00feb30 2023-07-14T19:02:36.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec6a75700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 0x7f5ec019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:36.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec6a75700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 0x7f5ec019f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34556/0 (socket says 172.21.15.78:34556) 2023-07-14T19:02:36.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec6a75700 1 -- 172.21.15.78:0/4279598952 learned_addr learned my addr 172.21.15.78:0/4279598952 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:36.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.676+0000 7f5ec6274700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec00feb30 0x7f5ec0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:36.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec5a73700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ec00ffd20 0x7f5ec0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:36.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec6a75700 1 -- 172.21.15.78:0/4279598952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec00feb30 msgr2=0x7f5ec0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec6a75700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec00feb30 0x7f5ec0197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec6a75700 1 -- 172.21.15.78:0/4279598952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ec00ffd20 msgr2=0x7f5ec0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec6a75700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ec00ffd20 0x7f5ec0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec6a75700 1 -- 172.21.15.78:0/4279598952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb8009d00 con 0x7f5ec01006b0 2023-07-14T19:02:36.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec6274700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec00feb30 0x7f5ec0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:36.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec5a73700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ec00ffd20 0x7f5ec0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:36.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec6a75700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 0x7f5ec019f0e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800bad0 tx=0x7f5eb800bbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:36.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5eb77fe700 1 -- 172.21.15.78:0/4279598952 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb800e040 con 0x7f5ec01006b0 2023-07-14T19:02:36.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec019fc00 con 0x7f5ec01006b0 2023-07-14T19:02:36.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5eb77fe700 1 -- 172.21.15.78:0/4279598952 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5eb8004e40 con 0x7f5ec01006b0 2023-07-14T19:02:36.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5eb77fe700 1 -- 172.21.15.78:0/4279598952 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb80053b0 con 0x7f5ec01006b0 2023-07-14T19:02:36.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.677+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec01a0080 con 0x7f5ec01006b0 2023-07-14T19:02:36.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.678+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec010a1c0 con 0x7f5ec01006b0 2023-07-14T19:02:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.679+0000 7f5eb77fe700 1 -- 172.21.15.78:0/4279598952 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5eb801da90 con 0x7f5ec01006b0 2023-07-14T19:02:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.679+0000 7f5eb77fe700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eac0a11a0 0x7f5eac0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.679+0000 7f5ec6274700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eac0a11a0 0x7f5eac0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.680+0000 7f5eb77fe700 1 -- 172.21.15.78:0/4279598952 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5eb80ce1c0 con 0x7f5ec01006b0 2023-07-14T19:02:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.680+0000 7f5ec6274700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eac0a11a0 0x7f5eac0a3650 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00ffb80 tx=0x7f5ebc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:36.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.684+0000 7f5eb77fe700 1 -- 172.21.15.78:0/4279598952 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5eb80977b0 con 0x7f5ec01006b0 2023-07-14T19:02:36.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.878+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ec01a0630 con 0x7f5ec01006b0 2023-07-14T19:02:36.883 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.879+0000 7f5eb77fe700 1 -- 172.21.15.78:0/4279598952 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7f5eb809add0 con 0x7f5ec01006b0 2023-07-14T19:02:36.884 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:36.884 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:36.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eac0a11a0 msgr2=0x7f5eac0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.885 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eac0a11a0 0x7f5eac0a3650 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00ffb80 tx=0x7f5ebc005f90 comp rx=0 tx=0).stop 2023-07-14T19:02:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 msgr2=0x7f5ec019f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 0x7f5ec019f0e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5eb800bad0 tx=0x7f5eb800bbb0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 shutdown_connections 2023-07-14T19:02:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eac0a11a0 0x7f5eac0a3650 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ec01006b0 0x7f5ec019f0e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ec00feb30 0x7f5ec0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 --2- 172.21.15.78:0/4279598952 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ec00ffd20 0x7f5ec0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.881+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 >> 172.21.15.78:0/4279598952 conn(0x7f5ec00fa150 msgr2=0x7f5ec0068a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.882+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 shutdown_connections 2023-07-14T19:02:36.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:36.882+0000 7f5ec84d8700 1 -- 172.21.15.78:0/4279598952 wait complete. 2023-07-14T19:02:36.916 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:37.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 -- 172.21.15.78:0/2375801489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d41006b0 msgr2=0x7f85d4106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 --2- 172.21.15.78:0/2375801489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d41006b0 0x7f85d4106f70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f85cc00b0a0 tx=0x7f85cc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 -- 172.21.15.78:0/2375801489 shutdown_connections 2023-07-14T19:02:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 --2- 172.21.15.78:0/2375801489 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d41006b0 0x7f85d4106f70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 --2- 172.21.15.78:0/2375801489 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85d40feb30 0x7f85d40fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 --2- 172.21.15.78:0/2375801489 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85d40ffd20 0x7f85d4100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 -- 172.21.15.78:0/2375801489 >> 172.21.15.78:0/2375801489 conn(0x7f85d40fa150 msgr2=0x7f85d40fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 -- 172.21.15.78:0/2375801489 shutdown_connections 2023-07-14T19:02:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 -- 172.21.15.78:0/2375801489 wait complete. 2023-07-14T19:02:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 Processor -- start 2023-07-14T19:02:37.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.034+0000 7f85db190700 1 -- start start 2023-07-14T19:02:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85db190700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d40feb30 0x7f85d4194830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85db190700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85d40ffd20 0x7f85d4194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85db190700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85d41006b0 0x7f85d419bde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85db190700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d419c3b0 con 0x7f85d40feb30 2023-07-14T19:02:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85db190700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d419c4f0 con 0x7f85d40ffd20 2023-07-14T19:02:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85db190700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d419c680 con 0x7f85d41006b0 2023-07-14T19:02:37.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d40feb30 0x7f85d4194830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d40feb30 0x7f85d4194830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34566/0 (socket says 172.21.15.78:34566) 2023-07-14T19:02:37.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 -- 172.21.15.78:0/2696436617 learned_addr learned my addr 172.21.15.78:0/2696436617 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:37.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d972d700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85d41006b0 0x7f85d419bde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 -- 172.21.15.78:0/2696436617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85d41006b0 msgr2=0x7f85d419bde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85d41006b0 0x7f85d419bde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 -- 172.21.15.78:0/2696436617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85d40ffd20 msgr2=0x7f85d4194d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d3fff700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85d40ffd20 0x7f85d4194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85d40ffd20 0x7f85d4194d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 -- 172.21.15.78:0/2696436617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85cc009d00 con 0x7f85d40feb30 2023-07-14T19:02:37.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d8f2c700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d40feb30 0x7f85d4194830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f85c800c970 tx=0x7f85c800cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d1ffb700 1 -- 172.21.15.78:0/2696436617 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8004220 con 0x7f85d40feb30 2023-07-14T19:02:37.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d1ffb700 1 -- 172.21.15.78:0/2696436617 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f85c8007d30 con 0x7f85d40feb30 2023-07-14T19:02:37.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d419c960 con 0x7f85d40feb30 2023-07-14T19:02:37.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.035+0000 7f85d1ffb700 1 -- 172.21.15.78:0/2696436617 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8003bf0 con 0x7f85d40feb30 2023-07-14T19:02:37.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.036+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d419cec0 con 0x7f85d40feb30 2023-07-14T19:02:37.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.037+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d4106020 con 0x7f85d40feb30 2023-07-14T19:02:37.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.037+0000 7f85d1ffb700 1 -- 172.21.15.78:0/2696436617 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f85c80078a0 con 0x7f85d40feb30 2023-07-14T19:02:37.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.038+0000 7f85d1ffb700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85c40a1220 0x7f85c40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.038+0000 7f85d3fff700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85c40a1220 0x7f85c40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.038+0000 7f85d1ffb700 1 -- 172.21.15.78:0/2696436617 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f85c80cc700 con 0x7f85d40feb30 2023-07-14T19:02:37.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.039+0000 7f85d3fff700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85c40a1220 0x7f85c40a36d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009af0 tx=0x7f85c00094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.046 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.043+0000 7f85d1ffb700 1 -- 172.21.15.78:0/2696436617 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f85c8095c60 con 0x7f85d40feb30 2023-07-14T19:02:37.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:36 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/461443891' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T19:02:37.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:36 smithi102 ceph-mon[129280]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:37.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:36 smithi102 ceph-mon[129280]: pgmap v258: 129 pgs: 110 active+clean, 1 creating+activating, 1 creating+peering, 17 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-14T19:02:37.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:36 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1444280253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:37.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:36 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2089451994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:37.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:36 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/461443891' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T19:02:37.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:36 smithi088 ceph-mon[128973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:37.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:36 smithi088 ceph-mon[128973]: pgmap v258: 129 pgs: 110 active+clean, 1 creating+activating, 1 creating+peering, 17 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-14T19:02:37.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:36 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1444280253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:37.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:36 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2089451994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.272+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85d402c8e0 con 0x7f85d40feb30 2023-07-14T19:02:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.273+0000 7f85d1ffb700 1 -- 172.21.15.78:0/2696436617 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+10748 (secure 0 0 0) 0x7f85c8099280 con 0x7f85d40feb30 2023-07-14T19:02:37.277 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:37.277 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":7},{"gid":24464,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":6},{"gid":24469,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":8},{"gid":24485,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":9},{"gid":24487,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:02:35.807625+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":24503},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2023-07-14T19:02:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85c40a1220 msgr2=0x7f85c40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85c40a1220 0x7f85c40a36d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f85c0009af0 tx=0x7f85c00094f0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d40feb30 msgr2=0x7f85d4194830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d40feb30 0x7f85d4194830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f85c800c970 tx=0x7f85c800cd30 comp rx=0 tx=0).stop 2023-07-14T19:02:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 shutdown_connections 2023-07-14T19:02:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85c40a1220 0x7f85c40a36d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85d41006b0 0x7f85d419bde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85d40feb30 0x7f85d4194830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 --2- 172.21.15.78:0/2696436617 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85d40ffd20 0x7f85d4194d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 >> 172.21.15.78:0/2696436617 conn(0x7f85d40fa150 msgr2=0x7f85d4103160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 shutdown_connections 2023-07-14T19:02:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.275+0000 7f85db190700 1 -- 172.21.15.78:0/2696436617 wait complete. 2023-07-14T19:02:37.281 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 17 2023-07-14T19:02:37.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:36 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/461443891' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2023-07-14T19:02:37.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:36 smithi078 ceph-mon[127425]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:37.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:36 smithi078 ceph-mon[127425]: pgmap v258: 129 pgs: 110 active+clean, 1 creating+activating, 1 creating+peering, 17 unknown; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 8 op/s 2023-07-14T19:02:37.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:36 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1444280253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:37.290 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:36 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2089451994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:37.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:37.411 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 -- 172.21.15.78:0/574254149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301058f0 msgr2=0x7f0e30107d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/574254149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301058f0 0x7f0e30107d50 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800b0a0 tx=0x7f0e2800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 -- 172.21.15.78:0/574254149 shutdown_connections 2023-07-14T19:02:37.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/574254149 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301058f0 0x7f0e30107d50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/574254149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e301006b0 0x7f0e30102a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/574254149 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e30102fd0 0x7f0e301053b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 -- 172.21.15.78:0/574254149 >> 172.21.15.78:0/574254149 conn(0x7f0e300fa150 msgr2=0x7f0e300fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 -- 172.21.15.78:0/574254149 shutdown_connections 2023-07-14T19:02:37.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.407+0000 7f0e37f7e700 1 -- 172.21.15.78:0/574254149 wait complete. 2023-07-14T19:02:37.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e37f7e700 1 Processor -- start 2023-07-14T19:02:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e37f7e700 1 -- start start 2023-07-14T19:02:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e37f7e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301006b0 0x7f0e30195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e37f7e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e30102fd0 0x7f0e30195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e37f7e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e301058f0 0x7f0e3019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e37f7e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3019d400 con 0x7f0e301006b0 2023-07-14T19:02:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e37f7e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3019d540 con 0x7f0e30102fd0 2023-07-14T19:02:37.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e37f7e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e3019d6d0 con 0x7f0e301058f0 2023-07-14T19:02:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e35d1a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301006b0 0x7f0e30195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e35d1a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301006b0 0x7f0e30195880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34580/0 (socket says 172.21.15.78:34580) 2023-07-14T19:02:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e35d1a700 1 -- 172.21.15.78:0/1596327008 learned_addr learned my addr 172.21.15.78:0/1596327008 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e35519700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e30102fd0 0x7f0e30195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.408+0000 7f0e3651b700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e301058f0 0x7f0e3019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e35d1a700 1 -- 172.21.15.78:0/1596327008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e301058f0 msgr2=0x7f0e3019ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e35d1a700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e301058f0 0x7f0e3019ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e35d1a700 1 -- 172.21.15.78:0/1596327008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e30102fd0 msgr2=0x7f0e30195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e35d1a700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e30102fd0 0x7f0e30195dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e35d1a700 1 -- 172.21.15.78:0/1596327008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e28009d00 con 0x7f0e301006b0 2023-07-14T19:02:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e35d1a700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301006b0 0x7f0e30195880 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000c9b0 tx=0x7f0e2000cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e26ffd700 1 -- 172.21.15.78:0/1596327008 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20007b00 con 0x7f0e301006b0 2023-07-14T19:02:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e3019d9b0 con 0x7f0e301006b0 2023-07-14T19:02:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e26ffd700 1 -- 172.21.15.78:0/1596327008 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0e20007c60 con 0x7f0e301006b0 2023-07-14T19:02:37.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e26ffd700 1 -- 172.21.15.78:0/1596327008 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e20018660 con 0x7f0e301006b0 2023-07-14T19:02:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.409+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e3019dec0 con 0x7f0e301006b0 2023-07-14T19:02:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.410+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e300fbcc0 con 0x7f0e301006b0 2023-07-14T19:02:37.418 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.411+0000 7f0e26ffd700 1 -- 172.21.15.78:0/1596327008 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f0e2000f450 con 0x7f0e301006b0 2023-07-14T19:02:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.411+0000 7f0e26ffd700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e1c0a11a0 0x7f0e1c0a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.411+0000 7f0e35519700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e1c0a11a0 0x7f0e1c0a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.412+0000 7f0e26ffd700 1 -- 172.21.15.78:0/1596327008 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f0e200cce50 con 0x7f0e301006b0 2023-07-14T19:02:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.412+0000 7f0e35519700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e1c0a11a0 0x7f0e1c0a3650 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c005950 tx=0x7f0e2c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.415+0000 7f0e26ffd700 1 -- 172.21.15.78:0/1596327008 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0e20096440 con 0x7f0e301006b0 2023-07-14T19:02:37.607 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.603+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e3002c9d0 con 0x7f0e301006b0 2023-07-14T19:02:37.608 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.605+0000 7f0e26ffd700 1 -- 172.21.15.78:0/1596327008 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7f0e20099a60 con 0x7f0e301006b0 2023-07-14T19:02:37.609 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:37.609 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:37.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e1c0a11a0 msgr2=0x7f0e1c0a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e1c0a11a0 0x7f0e1c0a3650 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c005950 tx=0x7f0e2c0058e0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301006b0 msgr2=0x7f0e30195880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301006b0 0x7f0e30195880 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0e2000c9b0 tx=0x7f0e2000cd70 comp rx=0 tx=0).stop 2023-07-14T19:02:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 shutdown_connections 2023-07-14T19:02:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e1c0a11a0 0x7f0e1c0a3650 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e301058f0 0x7f0e3019ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e301006b0 0x7f0e30195880 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 --2- 172.21.15.78:0/1596327008 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e30102fd0 0x7f0e30195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 >> 172.21.15.78:0/1596327008 conn(0x7f0e300fa150 msgr2=0x7f0e30101310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 shutdown_connections 2023-07-14T19:02:37.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.607+0000 7f0e37f7e700 1 -- 172.21.15.78:0/1596327008 wait complete. 2023-07-14T19:02:37.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-07-14T19:02:30.340765+0000', 'modified': '2023-07-14T19:02:35.807625+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24503}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6839/1379790346', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 1379790346}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 1379790346}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2023-07-14T19:02:37.623 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-07-14T19:02:37.624 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:02:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 -- 172.21.15.78:0/343926018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 msgr2=0x7fec48106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/343926018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 0x7fec48106f50 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fec38009990 tx=0x7fec38009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.745 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 -- 172.21.15.78:0/343926018 shutdown_connections 2023-07-14T19:02:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/343926018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec480feb10 0x7fec480fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/343926018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 0x7fec48106f50 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/343926018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec480ffd00 0x7fec48100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 -- 172.21.15.78:0/343926018 >> 172.21.15.78:0/343926018 conn(0x7fec480fa130 msgr2=0x7fec480fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 -- 172.21.15.78:0/343926018 shutdown_connections 2023-07-14T19:02:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 -- 172.21.15.78:0/343926018 wait complete. 2023-07-14T19:02:37.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.741+0000 7fec4dbf2700 1 Processor -- start 2023-07-14T19:02:37.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec4dbf2700 1 -- start start 2023-07-14T19:02:37.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec4dbf2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec480feb10 0x7fec48194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.748 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec4dbf2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec480ffd00 0x7fec48194c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec4dbf2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 0x7fec4819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec4dbf2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4819c280 con 0x7fec48100690 2023-07-14T19:02:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec4dbf2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4819c3c0 con 0x7fec480ffd00 2023-07-14T19:02:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec4dbf2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec4819c550 con 0x7fec480feb10 2023-07-14T19:02:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 0x7fec4819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 0x7fec4819bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34598/0 (socket says 172.21.15.78:34598) 2023-07-14T19:02:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 -- 172.21.15.78:0/2284214549 learned_addr learned my addr 172.21.15.78:0/2284214549 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:37.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec477fe700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec480feb10 0x7fec48194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec46ffd700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec480ffd00 0x7fec48194c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 -- 172.21.15.78:0/2284214549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec480feb10 msgr2=0x7fec48194700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec480feb10 0x7fec48194700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 -- 172.21.15.78:0/2284214549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec480ffd00 msgr2=0x7fec48194c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec480ffd00 0x7fec48194c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 -- 172.21.15.78:0/2284214549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380095f0 con 0x7fec48100690 2023-07-14T19:02:37.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec477fe700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec480feb10 0x7fec48194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:37.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec47fff700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 0x7fec4819bcb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fec38004730 tx=0x7fec38005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec44ff9700 1 -- 172.21.15.78:0/2284214549 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3801d070 con 0x7fec48100690 2023-07-14T19:02:37.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec4dbf2700 1 -- 172.21.15.78:0/2284214549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4819c7d0 con 0x7fec48100690 2023-07-14T19:02:37.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec44ff9700 1 -- 172.21.15.78:0/2284214549 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fec3800be90 con 0x7fec48100690 2023-07-14T19:02:37.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.742+0000 7fec44ff9700 1 -- 172.21.15.78:0/2284214549 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec380053b0 con 0x7fec48100690 2023-07-14T19:02:37.751 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.743+0000 7fec4dbf2700 1 -- 172.21.15.78:0/2284214549 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4819cc50 con 0x7fec48100690 2023-07-14T19:02:37.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.745+0000 7fec44ff9700 1 -- 172.21.15.78:0/2284214549 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fec38004810 con 0x7fec48100690 2023-07-14T19:02:37.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.745+0000 7fec44ff9700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fec340a11d0 0x7fec340a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.745+0000 7fec44ff9700 1 -- 172.21.15.78:0/2284214549 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fec380cdd30 con 0x7fec48100690 2023-07-14T19:02:37.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.745+0000 7fec477fe700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fec340a11d0 0x7fec340a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec477fe700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fec340a11d0 0x7fec340a3680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fec480ffb60 tx=0x7fec40009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec59d8ab80 1 Processor -- start 2023-07-14T19:02:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec59d8ab80 1 -- start start 2023-07-14T19:02:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f21254e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f21260c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f2126ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f2027c60 con 0x5587f2125a20 2023-07-14T19:02:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f207a9f0 con 0x5587f21250d0 2023-07-14T19:02:37.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f2073e80 con 0x5587f2126650 2023-07-14T19:02:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.746+0000 7fec46ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f21260c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec46ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f21260c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34604/0 (socket says 172.21.15.78:34604) 2023-07-14T19:02:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec46ffd700 1 -- 172.21.15.78:0/1428495586 learned_addr learned my addr 172.21.15.78:0/1428495586 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec477fe700 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f21254e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec47fff700 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f2126ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec46ffd700 1 -- 172.21.15.78:0/1428495586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 msgr2=0x5587f2126ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:02:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec46ffd700 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f2126ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec46ffd700 1 -- 172.21.15.78:0/1428495586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 msgr2=0x5587f21254e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:02:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec46ffd700 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f21254e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec46ffd700 1 -- 172.21.15.78:0/1428495586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587f1ea1d90 con 0x5587f2125a20 2023-07-14T19:02:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec47fff700 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f2126ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec46ffd700 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f21260c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec30009ce0 tx=0x7fec30010370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31a7e6f645b99cfd server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec3dffb700 1 -- 172.21.15.78:0/1428495586 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec300097b0 con 0x5587f2125a20 2023-07-14T19:02:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec3dffb700 1 -- 172.21.15.78:0/1428495586 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fec300044b0 con 0x5587f2125a20 2023-07-14T19:02:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.747+0000 7fec3dffb700 1 -- 172.21.15.78:0/1428495586 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30003dd0 con 0x5587f2125a20 2023-07-14T19:02:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 -- 172.21.15.78:0/1428495586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 msgr2=0x5587f21260c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f21260c0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fec30009ce0 tx=0x7fec30010370 comp rx=0 tx=0).stop 2023-07-14T19:02:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 -- 172.21.15.78:0/1428495586 shutdown_connections 2023-07-14T19:02:37.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f21254e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f2126ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/1428495586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f21260c0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 -- 172.21.15.78:0/1428495586 >> 172.21.15.78:0/1428495586 conn(0x5587f2118630 msgr2=0x5587f211d980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 -- 172.21.15.78:0/1428495586 shutdown_connections 2023-07-14T19:02:37.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 -- 172.21.15.78:0/1428495586 wait complete. 2023-07-14T19:02:37.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 -- start start 2023-07-14T19:02:37.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.748+0000 7fec59d8ab80 1 Processor -- start 2023-07-14T19:02:37.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f212eb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 0x5587f212f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2136080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f2013a70 con 0x5587f2126650 2023-07-14T19:02:37.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f2013770 con 0x5587f21250d0 2023-07-14T19:02:37.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f1ef78d0 con 0x5587f2125a20 2023-07-14T19:02:37.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 10 client.0 _ll_register_callbacks cb 0x5587f2011710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:02:37.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2136080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2136080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34612/0 (socket says 172.21.15.78:34612) 2023-07-14T19:02:37.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 -- 172.21.15.78:0/4155892586 learned_addr learned my addr 172.21.15.78:0/4155892586 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:37.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec46ffd700 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 0x5587f212f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec477fe700 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f212eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 -- 172.21.15.78:0/4155892586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 msgr2=0x5587f212f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:02:37.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 0x5587f212f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 -- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 msgr2=0x5587f212eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:02:37.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f212eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 -- 172.21.15.78:0/4155892586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec30009910 con 0x5587f2126650 2023-07-14T19:02:37.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec477fe700 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f212eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:37.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec47fff700 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2136080 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fec380979f0 tx=0x7fec38004dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58eb4d2a1c826f98 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec26ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:02:37.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec26ffd700 1 -- 172.21.15.78:0/4155892586 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3809b3e0 con 0x5587f2126650 2023-07-14T19:02:37.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec26ffd700 1 -- 172.21.15.78:0/4155892586 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fec3809a470 con 0x5587f2126650 2023-07-14T19:02:37.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 1 -- 172.21.15.78:0/4155892586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5587f1fcd4a0 con 0x5587f2126650 2023-07-14T19:02:37.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec26ffd700 1 -- 172.21.15.78:0/4155892586 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec3809a630 con 0x5587f2126650 2023-07-14T19:02:37.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 10 client.14745 fetch_fsmap learned FSMap version 17 2023-07-14T19:02:37.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.749+0000 7fec59d8ab80 1 -- 172.21.15.78:0/4155892586 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5587f1e8ac10 con 0x5587f2126650 2023-07-14T19:02:37.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec26ffd700 1 -- 172.21.15.78:0/4155892586 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5587f1fcd4a0 con 0x5587f2126650 2023-07-14T19:02:37.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec26ffd700 1 -- 172.21.15.78:0/4155892586 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fec38095b90 con 0x5587f2126650 2023-07-14T19:02:37.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec59d8ab80 10 client.14745 fetch_fsmap finished waiting for FSMap version 17 2023-07-14T19:02:37.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec59d8ab80 10 client.14745 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:02:37.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec59d8ab80 20 client.14745 populate_metadata read hostname 'smithi078' 2023-07-14T19:02:37.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x5587f2139950 0x5587f2138f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec59d8ab80 4 client.14745 mds_command: new command op to 24503 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:02:37.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec59d8ab80 1 -- 172.21.15.78:0/4155892586 --> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5587f20f1d50 con 0x5587f2139950 2023-07-14T19:02:37.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec477fe700 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x5587f2139950 0x5587f2138f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec26ffd700 10 client.14745 ms_handle_connect on v2:172.21.15.88:6838/1379790346 2023-07-14T19:02:37.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.750+0000 7fec477fe700 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x5587f2139950 0x5587f2138f20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f45c213fc7a9294a server_cookie=e51079fc2532f3f1 in_seq=0 out_seq=0 2023-07-14T19:02:37.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec26ffd700 1 -- 172.21.15.78:0/4155892586 <== mds.0 v2:172.21.15.88:6838/1379790346 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5587f20f1d50 con 0x5587f2139950 2023-07-14T19:02:37.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec26ffd700 10 client.14745 handle_command_reply: tid=0 2023-07-14T19:02:37.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 client.14745 shutdown 2023-07-14T19:02:37.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 -- 172.21.15.78:0/4155892586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 msgr2=0x5587f2136080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2136080 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fec380979f0 tx=0x7fec38004dd0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 -- 172.21.15.78:0/4155892586 shutdown_connections 2023-07-14T19:02:37.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x5587f2139950 0x5587f2138f20 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f212eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2136080 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/4155892586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 0x5587f212f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec26ffd700 0 client.14745 ms_handle_reset on v2:172.21.15.88:6838/1379790346 2023-07-14T19:02:37.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 -- 172.21.15.78:0/4155892586 >> 172.21.15.78:0/4155892586 conn(0x5587f2118630 msgr2=0x5587f211c600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 -- 172.21.15.78:0/4155892586 shutdown_connections 2023-07-14T19:02:37.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 1 -- 172.21.15.78:0/4155892586 wait complete. 2023-07-14T19:02:37.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.751+0000 7fec59d8ab80 20 client.14745 trim_cache size 0 max 16384 2023-07-14T19:02:37.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec59d8ab80 1 Processor -- start 2023-07-14T19:02:37.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec59d8ab80 1 -- start start 2023-07-14T19:02:37.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f2136910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f2136040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f212ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f2042790 con 0x5587f2125a20 2023-07-14T19:02:37.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f1f03de0 con 0x5587f21250d0 2023-07-14T19:02:37.774 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f200fd50 con 0x5587f2126650 2023-07-14T19:02:37.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f2136040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f2136040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34618/0 (socket says 172.21.15.78:34618) 2023-07-14T19:02:37.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 -- 172.21.15.78:0/2401745725 learned_addr learned my addr 172.21.15.78:0/2401745725 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:37.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec47fff700 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f212ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec477fe700 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f2136910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 -- 172.21.15.78:0/2401745725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 msgr2=0x5587f212ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:02:37.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f212ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 -- 172.21.15.78:0/2401745725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 msgr2=0x5587f2136910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:02:37.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f2136910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 -- 172.21.15.78:0/2401745725 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5587f1ec8200 con 0x5587f2125a20 2023-07-14T19:02:37.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec46ffd700 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f2136040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec30009cb0 tx=0x7fec30005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56028988291df9c8 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.769+0000 7fec3e7fc700 1 -- 172.21.15.78:0/2401745725 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec300097b0 con 0x5587f2125a20 2023-07-14T19:02:37.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec3e7fc700 1 -- 172.21.15.78:0/2401745725 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fec30022040 con 0x5587f2125a20 2023-07-14T19:02:37.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec3e7fc700 1 -- 172.21.15.78:0/2401745725 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec30004b10 con 0x5587f2125a20 2023-07-14T19:02:37.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 -- 172.21.15.78:0/2401745725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 msgr2=0x5587f2136040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f2136040 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fec30009cb0 tx=0x7fec30005dc0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 -- 172.21.15.78:0/2401745725 shutdown_connections 2023-07-14T19:02:37.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f2136910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2126650 0x5587f212ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/2401745725 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2125a20 0x5587f2136040 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 -- 172.21.15.78:0/2401745725 >> 172.21.15.78:0/2401745725 conn(0x5587f2118630 msgr2=0x5587f212d870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 -- 172.21.15.78:0/2401745725 shutdown_connections 2023-07-14T19:02:37.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 -- 172.21.15.78:0/2401745725 wait complete. 2023-07-14T19:02:37.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.770+0000 7fec59d8ab80 1 -- start start 2023-07-14T19:02:37.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 1 Processor -- start 2023-07-14T19:02:37.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f21312b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 0x5587f21317f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2134600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f2134b40 con 0x5587f2126650 2023-07-14T19:02:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f2134cb0 con 0x5587f21250d0 2023-07-14T19:02:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5587f2134e70 con 0x5587f2125a20 2023-07-14T19:02:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 10 client.0 _ll_register_callbacks cb 0x7fec3809a8e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:02:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2134600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2134600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34626/0 (socket says 172.21.15.78:34626) 2023-07-14T19:02:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec46ffd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 0x5587f21317f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 -- 172.21.15.78:0/3322297037 learned_addr learned my addr 172.21.15.78:0/3322297037 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec477fe700 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f21312b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 -- 172.21.15.78:0/3322297037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 msgr2=0x5587f21317f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:02:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 0x5587f21317f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 -- 172.21.15.78:0/3322297037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 msgr2=0x5587f21312b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:02:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f21312b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 -- 172.21.15.78:0/3322297037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5587f1ea1d90 con 0x5587f2126650 2023-07-14T19:02:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec47fff700 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2134600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec380992d0 tx=0x7fec3809a3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980f6ba4588cb292 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec26ffd700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:02:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec26ffd700 1 -- 172.21.15.78:0/3322297037 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec3809a470 con 0x5587f2126650 2023-07-14T19:02:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec26ffd700 1 -- 172.21.15.78:0/3322297037 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fec3809a5d0 con 0x5587f2126650 2023-07-14T19:02:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.771+0000 7fec59d8ab80 1 -- 172.21.15.78:0/3322297037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec300044b0 con 0x5587f2126650 2023-07-14T19:02:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec26ffd700 1 -- 172.21.15.78:0/3322297037 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec380cb330 con 0x5587f2126650 2023-07-14T19:02:37.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec59d8ab80 10 client.14751 fetch_fsmap learned FSMap version 17 2023-07-14T19:02:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec59d8ab80 1 -- 172.21.15.78:0/3322297037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) v3 -- 0x5587f1ea0260 con 0x5587f2126650 2023-07-14T19:02:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec26ffd700 1 -- 172.21.15.78:0/3322297037 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fec38064080 con 0x5587f2126650 2023-07-14T19:02:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec26ffd700 1 -- 172.21.15.78:0/3322297037 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 17) v1 ==== 7482+0+0 (secure 0 0 0) 0x7fec3809b1e0 con 0x5587f2126650 2023-07-14T19:02:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec59d8ab80 10 client.14751 fetch_fsmap finished waiting for FSMap version 17 2023-07-14T19:02:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec59d8ab80 10 client.14751 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:02:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec59d8ab80 20 client.14751 populate_metadata read hostname 'smithi078' 2023-07-14T19:02:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x5587f21925c0 0x5587f218f600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:37.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec59d8ab80 4 client.14751 mds_command: new command op to 24503 tid=0[{"prefix": "status"}] 2023-07-14T19:02:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec59d8ab80 1 -- 172.21.15.78:0/3322297037 --> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5587f206ce60 con 0x5587f21925c0 2023-07-14T19:02:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.772+0000 7fec477fe700 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x5587f21925c0 0x5587f218f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec26ffd700 10 client.14751 ms_handle_connect on v2:172.21.15.88:6838/1379790346 2023-07-14T19:02:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec477fe700 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x5587f21925c0 0x5587f218f600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=33716a8e685070c2 server_cookie=7117fba576b62a97 in_seq=0 out_seq=0 2023-07-14T19:02:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec26ffd700 1 -- 172.21.15.78:0/3322297037 <== mds.0 v2:172.21.15.88:6838/1379790346 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x5587f206ce60 con 0x5587f21925c0 2023-07-14T19:02:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec26ffd700 10 client.14751 handle_command_reply: tid=0 2023-07-14T19:02:37.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 client.14751 shutdown 2023-07-14T19:02:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 -- 172.21.15.78:0/3322297037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 msgr2=0x5587f2134600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2134600 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fec380992d0 tx=0x7fec3809a3a0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 -- 172.21.15.78:0/3322297037 shutdown_connections 2023-07-14T19:02:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x5587f21925c0 0x5587f218f600 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5587f21250d0 0x5587f21312b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5587f2126650 0x5587f2134600 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 --2- 172.21.15.78:0/3322297037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5587f2125a20 0x5587f21317f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec26ffd700 0 client.14751 ms_handle_reset on v2:172.21.15.88:6838/1379790346 2023-07-14T19:02:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 -- 172.21.15.78:0/3322297037 >> 172.21.15.78:0/3322297037 conn(0x5587f2118630 msgr2=0x5587f212ca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 -- 172.21.15.78:0/3322297037 shutdown_connections 2023-07-14T19:02:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 1 -- 172.21.15.78:0/3322297037 wait complete. 2023-07-14T19:02:37.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec59d8ab80 20 client.14751 trim_cache size 0 max 16384 2023-07-14T19:02:37.788 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:02:37.789 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T19:02:37.789 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-07-14T19:02:37.789 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24503, 2023-07-14T19:02:37.789 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-07-14T19:02:37.789 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-07-14T19:02:37.789 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-07-14T19:02:37.789 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 6.8450448880000003, 2023-07-14T19:02:37.790 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 17, 2023-07-14T19:02:37.790 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 85, 2023-07-14T19:02:37.790 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T19:02:37.790 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 12.856799918 2023-07-14T19:02:37.790 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:02:37.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.773+0000 7fec4dbf2700 1 -- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fec340a11d0 msgr2=0x7fec340a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fec340a11d0 0x7fec340a3680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fec480ffb60 tx=0x7fec40009040 comp rx=0 tx=0).stop 2023-07-14T19:02:37.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 -- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 msgr2=0x7fec4819bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:37.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 0x7fec4819bcb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fec38004730 tx=0x7fec38005ef0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 -- 172.21.15.78:0/2284214549 shutdown_connections 2023-07-14T19:02:37.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fec340a11d0 0x7fec340a3680 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fec48100690 0x7fec4819bcb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec480feb10 0x7fec48194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 --2- 172.21.15.78:0/2284214549 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fec480ffd00 0x7fec48194c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:37.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 -- 172.21.15.78:0/2284214549 >> 172.21.15.78:0/2284214549 conn(0x7fec480fa130 msgr2=0x7fec480fc310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:37.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 -- 172.21.15.78:0/2284214549 shutdown_connections 2023-07-14T19:02:37.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:37.774+0000 7fec4dbf2700 1 -- 172.21.15.78:0/2284214549 wait complete. 2023-07-14T19:02:37.793 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T19:02:37.794 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-14T19:02:37.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T19:02:38.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:37 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/4279598952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:37 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2696436617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.048 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:37 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1596327008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:37 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/4279598952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:37 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2696436617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:37 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1596327008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:37 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/4279598952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:37 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2696436617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:37 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1596327008' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:38.342 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:02:39.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:38 smithi102 ceph-mon[129280]: pgmap v259: 129 pgs: 127 active+clean, 1 creating+activating, 1 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2023-07-14T19:02:39.152 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:38 smithi078 ceph-mon[127425]: pgmap v259: 129 pgs: 127 active+clean, 1 creating+activating, 1 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2023-07-14T19:02:39.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:38 smithi088 ceph-mon[128973]: pgmap v259: 129 pgs: 127 active+clean, 1 creating+activating, 1 creating+peering; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2023-07-14T19:02:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 -- 172.21.15.78:0/3955498484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b4103b60 msgr2=0x7fe9b4103f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 --2- 172.21.15.78:0/3955498484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b4103b60 0x7fe9b4103f30 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00099c0 tx=0x7fe9b0009cd0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 -- 172.21.15.78:0/3955498484 shutdown_connections 2023-07-14T19:02:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 --2- 172.21.15.78:0/3955498484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9b40ffd50 0x7fe9b410b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 --2- 172.21.15.78:0/3955498484 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b4103b60 0x7fe9b4103f30 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 --2- 172.21.15.78:0/3955498484 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9b40ff3a0 0x7fe9b40ff810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 -- 172.21.15.78:0/3955498484 >> 172.21.15.78:0/3955498484 conn(0x7fe9b4075280 msgr2=0x7fe9b4075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 -- 172.21.15.78:0/3955498484 shutdown_connections 2023-07-14T19:02:39.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 -- 172.21.15.78:0/3955498484 wait complete. 2023-07-14T19:02:39.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 Processor -- start 2023-07-14T19:02:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.521+0000 7fe9bb629700 1 -- start start 2023-07-14T19:02:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9bb629700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9b40ff3a0 0x7fe9b4100880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9bb629700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b40ffd50 0x7fe9b4100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9bb629700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9b4103b60 0x7fe9b41a3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9bb629700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b41016b0 con 0x7fe9b40ffd50 2023-07-14T19:02:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9bb629700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b41a43f0 con 0x7fe9b40ff3a0 2023-07-14T19:02:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9bb629700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b41a4560 con 0x7fe9b4103b60 2023-07-14T19:02:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b40ffd50 0x7fe9b4100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b40ffd50 0x7fe9b4100dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34636/0 (socket says 172.21.15.78:34636) 2023-07-14T19:02:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 -- 172.21.15.78:0/2529670287 learned_addr learned my addr 172.21.15.78:0/2529670287 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b9bc6700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9b4103b60 0x7fe9b41a3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 -- 172.21.15.78:0/2529670287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9b4103b60 msgr2=0x7fe9b41a3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9b4103b60 0x7fe9b41a3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b93c5700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9b40ff3a0 0x7fe9b4100880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 -- 172.21.15.78:0/2529670287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9b40ff3a0 msgr2=0x7fe9b4100880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9b40ff3a0 0x7fe9b4100880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 -- 172.21.15.78:0/2529670287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b00095f0 con 0x7fe9b40ffd50 2023-07-14T19:02:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b9bc6700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9b4103b60 0x7fe9b41a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b93c5700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9b40ff3a0 0x7fe9b4100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9b8bc4700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b40ffd50 0x7fe9b4100dc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400c9b0 tx=0x7fe9a400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9aa7fc700 1 -- 172.21.15.78:0/2529670287 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4007b00 con 0x7fe9b40ffd50 2023-07-14T19:02:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9aa7fc700 1 -- 172.21.15.78:0/2529670287 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe9a4007c60 con 0x7fe9b40ffd50 2023-07-14T19:02:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b41a47e0 con 0x7fe9b40ffd50 2023-07-14T19:02:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9aa7fc700 1 -- 172.21.15.78:0/2529670287 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4018660 con 0x7fe9b40ffd50 2023-07-14T19:02:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.522+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b41a4d30 con 0x7fe9b40ffd50 2023-07-14T19:02:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.523+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b404e340 con 0x7fe9b40ffd50 2023-07-14T19:02:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.524+0000 7fe9aa7fc700 1 -- 172.21.15.78:0/2529670287 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe9a400f450 con 0x7fe9b40ffd50 2023-07-14T19:02:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.525+0000 7fe9aa7fc700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe9a00a11d0 0x7fe9a00a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.525+0000 7fe9b93c5700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe9a00a11d0 0x7fe9a00a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.525+0000 7fe9aa7fc700 1 -- 172.21.15.78:0/2529670287 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe9a40ccb10 con 0x7fe9b40ffd50 2023-07-14T19:02:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.525+0000 7fe9b93c5700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe9a00a11d0 0x7fe9a00a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006010 tx=0x7fe9b0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.529+0000 7fe9aa7fc700 1 -- 172.21.15.78:0/2529670287 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe9a4096070 con 0x7fe9b40ffd50 2023-07-14T19:02:39.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.756+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe9b4101db0 con 0x7fe9b40ffd50 2023-07-14T19:02:39.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.756+0000 7fe9aa7fc700 1 -- 172.21.15.78:0/2529670287 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) v1 ==== 53+0+88 (secure 0 0 0) 0x7fe9a4099690 con 0x7fe9b40ffd50 2023-07-14T19:02:39.761 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-14T19:02:39.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.758+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe9a00a11d0 msgr2=0x7fe9a00a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:39.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.758+0000 7fe9bb629700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe9a00a11d0 0x7fe9a00a3680 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0006010 tx=0x7fe9b0000bc0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.758+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b40ffd50 msgr2=0x7fe9b4100dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:39.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.758+0000 7fe9bb629700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b40ffd50 0x7fe9b4100dc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400c9b0 tx=0x7fe9a400cd70 comp rx=0 tx=0).stop 2023-07-14T19:02:39.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.759+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 shutdown_connections 2023-07-14T19:02:39.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.759+0000 7fe9bb629700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe9a00a11d0 0x7fe9a00a3680 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.759+0000 7fe9bb629700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe9b40ff3a0 0x7fe9b4100880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.763 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.759+0000 7fe9bb629700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9b4103b60 0x7fe9b41a3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.759+0000 7fe9bb629700 1 --2- 172.21.15.78:0/2529670287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe9b40ffd50 0x7fe9b4100dc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:39.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.759+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 >> 172.21.15.78:0/2529670287 conn(0x7fe9b4075280 msgr2=0x7fe9b40fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:39.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.759+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 shutdown_connections 2023-07-14T19:02:39.764 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:39.759+0000 7fe9bb629700 1 -- 172.21.15.78:0/2529670287 wait complete. 2023-07-14T19:02:40.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:39 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2529670287' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:02:40.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:39 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2529670287' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:02:40.181 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-07-14T19:02:40.181 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T19:02:40.181 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2023-07-14T19:02:40.182 INFO:tasks.cephfs.mount:self.client.name = client.admin 2023-07-14T19:02:40.182 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2023-07-14T19:02:40.182 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-07-14T19:02:40.182 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T19:02:40.183 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T19:02:40.183 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2023-07-14T19:02:40.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.184 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2023-07-14T19:02:40.202 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T19:02:40.202 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T19:02:40.203 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T19:02:40.203 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:02:40.203 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2023-07-14T19:02:40.203 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:02:40.203 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:02:40.203 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:02:40.204 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:02:40.204 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:02:40.204 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:02:40.204 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2023-07-14T19:02:40.204 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T19:02:40.204 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2023-07-14T19:02:40.204 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T19:02:40.204 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 41505sec preferred_lft 41505sec 2023-07-14T19:02:40.205 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::8e6d:82b2:36a8:5b8d/64 scope link noprefixroute 2023-07-14T19:02:40.205 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:02:40.205 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T19:02:40.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.206 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:02:40.207 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:02:40.207 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2023-07-14T19:02:40.207 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2023-07-14T19:02:40.208 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2023-07-14T19:02:40.208 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T19:02:40.208 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:02:40.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:39 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2529670287' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:02:40.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.375 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T19:02:40.403 INFO:teuthology.orchestra.run.smithi078.stdout:1 2023-07-14T19:02:40.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.404 DEBUG:teuthology.orchestra.run.smithi078:> route 2023-07-14T19:02:40.472 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2023-07-14T19:02:40.472 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T19:02:40.472 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T19:02:40.472 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T19:02:40.472 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T19:02:40.473 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.473 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:02:40.474 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:02:40.474 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T19:02:40.475 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T19:02:40.475 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T19:02:40.476 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:02:40.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.766 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-07-14T19:02:40.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.784 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2023-07-14T19:02:40.842 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.842 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:02:40.842 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:02:40.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-14T19:02:40.847 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2023-07-14T19:02:40.847 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:02:40.925 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:40 smithi078 ceph-mon[127425]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2023-07-14T19:02:40.938 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2023-07-14T19:02:40.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:40.938 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:02:40.939 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:02:40.939 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2023-07-14T19:02:40.939 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T19:02:40.940 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2023-07-14T19:02:40.940 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2023-07-14T19:02:40.940 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2023-07-14T19:02:40.940 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:02:41.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:40 smithi102 ceph-mon[129280]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2023-07-14T19:02:41.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:41.145 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:02:41.145 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:02:41.147 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2023-07-14T19:02:41.148 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T19:02:41.148 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:02:41.212 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:40 smithi088 ceph-mon[128973]: pgmap v260: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 5 op/s 2023-07-14T19:02:41.246 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2023-07-14T19:02:41.247 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:02:41.247 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2023-07-14T19:02:41.264 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2023-07-14T19:02:41.265 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:02:41.265 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2023-07-14T19:02:41.322 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2023-07-14T19:02:41.389 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T19:02:41.389 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T19:02:41.455 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T19:02:41.456 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T19:02:41.457 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:41.457 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2023-07-14T19:02:41.514 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-14T19:02:41.515 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2023-07-14T19:02:41.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2023-07-14T19:02:41.583 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T19:02:41.584 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T19:02:41.650 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T19:02:41.651 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T19:02:41.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:41.652 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2023-07-14T19:02:41.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:2023-07-14T19:02:41.689+0000 7fe9f08835c0 -1 init, newargv = 0x55fe76539bc0 newargc=15 2023-07-14T19:02:41.693 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[164624]: starting ceph client 2023-07-14T19:02:41.698 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[164624]: starting fuse 2023-07-14T19:02:41.708 INFO:teuthology.orchestra.run.smithi078.stdout:124 2023-07-14T19:02:41.708 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-07-14T19:02:41.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-14T19:02:41.709 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2023-07-14T19:02:41.709 DEBUG:teuthology.orchestra.run.smithi078:> import re 2023-07-14T19:02:41.710 DEBUG:teuthology.orchestra.run.smithi078:> import os 2023-07-14T19:02:41.710 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2023-07-14T19:02:41.710 DEBUG:teuthology.orchestra.run.smithi078:> 2023-07-14T19:02:41.711 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2023-07-14T19:02:41.711 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2023-07-14T19:02:41.711 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2023-07-14T19:02:41.712 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2023-07-14T19:02:41.712 DEBUG:teuthology.orchestra.run.smithi078:> 2023-07-14T19:02:41.712 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2023-07-14T19:02:41.713 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2023-07-14T19:02:41.713 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2023-07-14T19:02:41.713 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2023-07-14T19:02:41.714 DEBUG:teuthology.orchestra.run.smithi078:> 2023-07-14T19:02:41.714 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2023-07-14T19:02:41.715 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-14T19:02:41.715 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-14T19:02:41.715 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-14T19:02:41.715 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2023-07-14T19:02:41.716 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2023-07-14T19:02:41.716 DEBUG:teuthology.orchestra.run.smithi078:> return f 2023-07-14T19:02:41.716 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-14T19:02:41.717 DEBUG:teuthology.orchestra.run.smithi078:> 2023-07-14T19:02:41.717 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.admin")) 2023-07-14T19:02:41.717 DEBUG:teuthology.orchestra.run.smithi078:> ' 2023-07-14T19:02:41.832 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.admin.164624.asok 2023-07-14T19:02:42.015 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.164624.asok 2023-07-14T19:02:42.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:42.015 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.164624.asok status 2023-07-14T19:02:42.172 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:02:42.172 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2023-07-14T19:02:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "81c9387d2e7576da065747ec9c16d1c7d69016ec", 2023-07-14T19:02:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:02:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "admin", 2023-07-14T19:02:42.173 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2023-07-14T19:02:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2023-07-14T19:02:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "164624", 2023-07-14T19:02:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/" 2023-07-14T19:02:42.174 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-14T19:02:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2023-07-14T19:02:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2023-07-14T19:02:42.175 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 14763, 2023-07-14T19:02:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2023-07-14T19:02:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2023-07-14T19:02:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2023-07-14T19:02:42.176 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 14763 2023-07-14T19:02:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-14T19:02:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2023-07-14T19:02:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2023-07-14T19:02:42.177 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2023-07-14T19:02:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1421371769 2023-07-14T19:02:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-14T19:02:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-14T19:02:42.178 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2023-07-14T19:02:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2023-07-14T19:02:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2023-07-14T19:02:42.179 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1421371769 2023-07-14T19:02:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-14T19:02:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.14763 192.168.0.1:0/1421371769", 2023-07-14T19:02:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/1421371769", 2023-07-14T19:02:42.180 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 1, 2023-07-14T19:02:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 17, 2023-07-14T19:02:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 85, 2023-07-14T19:02:42.181 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2023-07-14T19:02:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2023-07-14T19:02:42.182 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2023-07-14T19:02:42.182 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:02:42.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:42.183 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2023-07-14T19:02:42.184 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.admin 2023-07-14T19:02:42.201 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2023-07-14T19:02:42.203 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2023-07-14T19:02:42.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:42.205 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2023-07-14T19:02:42.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:42.269 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2023-07-14T19:02:42.376 INFO:teuthology.orchestra.run.smithi078.stdout:# file: . 2023-07-14T19:02:42.376 INFO:teuthology.orchestra.run.smithi078.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2023-07-14T19:02:42.376 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:42.377 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T19:02:42.377 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:02:42.394 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:42.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 -- 172.21.15.78:0/544075198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 msgr2=0x7fd1680fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:42.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 --2- 172.21.15.78:0/544075198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 0x7fd1680fef60 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fd16000b0a0 tx=0x7fd16000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 -- 172.21.15.78:0/544075198 shutdown_connections 2023-07-14T19:02:42.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 --2- 172.21.15.78:0/544075198 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1681006d0 0x7fd168106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 --2- 172.21.15.78:0/544075198 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 0x7fd1680fef60 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 --2- 172.21.15.78:0/544075198 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680ffd40 0x7fd168100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 -- 172.21.15.78:0/544075198 >> 172.21.15.78:0/544075198 conn(0x7fd1680fa150 msgr2=0x7fd1680fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:42.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 -- 172.21.15.78:0/544075198 shutdown_connections 2023-07-14T19:02:42.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.550+0000 7fd1701a0700 1 -- 172.21.15.78:0/544075198 wait complete. 2023-07-14T19:02:42.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd1701a0700 1 Processor -- start 2023-07-14T19:02:42.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd1701a0700 1 -- start start 2023-07-14T19:02:42.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd1701a0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 0x7fd1680692c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:42.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd1701a0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680ffd40 0x7fd16819a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:42.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd1701a0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1681006d0 0x7fd16819ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:42.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd1701a0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16819d4c0 con 0x7fd1680feb50 2023-07-14T19:02:42.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd1701a0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16819d630 con 0x7fd1680ffd40 2023-07-14T19:02:42.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd1701a0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16819d7f0 con 0x7fd1681006d0 2023-07-14T19:02:42.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 0x7fd1680692c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:42.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 0x7fd1680692c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34660/0 (socket says 172.21.15.78:34660) 2023-07-14T19:02:42.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 -- 172.21.15.78:0/1513207146 learned_addr learned my addr 172.21.15.78:0/1513207146 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:42.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16d73b700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680ffd40 0x7fd16819a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:42.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 -- 172.21.15.78:0/1513207146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1681006d0 msgr2=0x7fd16819ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:42.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1681006d0 0x7fd16819ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 -- 172.21.15.78:0/1513207146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680ffd40 msgr2=0x7fd16819a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:42.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680ffd40 0x7fd16819a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 -- 172.21.15.78:0/1513207146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd160009d00 con 0x7fd1680feb50 2023-07-14T19:02:42.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.551+0000 7fd16df3c700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 0x7fd1680692c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd160004d10 tx=0x7fd160004df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:42.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.552+0000 7fd15effd700 1 -- 172.21.15.78:0/1513207146 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd16000e040 con 0x7fd1680feb50 2023-07-14T19:02:42.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.552+0000 7fd15effd700 1 -- 172.21.15.78:0/1513207146 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd160006060 con 0x7fd1680feb50 2023-07-14T19:02:42.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.552+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16819da70 con 0x7fd1680feb50 2023-07-14T19:02:42.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.552+0000 7fd15effd700 1 -- 172.21.15.78:0/1513207146 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd160005870 con 0x7fd1680feb50 2023-07-14T19:02:42.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.552+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16819df70 con 0x7fd1680feb50 2023-07-14T19:02:42.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.553+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd168105ff0 con 0x7fd1680feb50 2023-07-14T19:02:42.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.554+0000 7fd15effd700 1 -- 172.21.15.78:0/1513207146 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd160005040 con 0x7fd1680feb50 2023-07-14T19:02:42.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.554+0000 7fd15effd700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd1540a11a0 0x7fd1540a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:42.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.554+0000 7fd15effd700 1 -- 172.21.15.78:0/1513207146 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd1600ce8e0 con 0x7fd1680feb50 2023-07-14T19:02:42.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.554+0000 7fd16d73b700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd1540a11a0 0x7fd1540a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:42.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.555+0000 7fd16d73b700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd1540a11a0 0x7fd1540a3650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd158005ea0 tx=0x7fd158005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:42.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.559+0000 7fd15effd700 1 -- 172.21.15.78:0/1513207146 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd160097ed0 con 0x7fd1680feb50 2023-07-14T19:02:42.756 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.752+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd16802c8e0 con 0x7fd1680feb50 2023-07-14T19:02:42.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.753+0000 7fd15effd700 1 -- 172.21.15.78:0/1513207146 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7fd16009b4f0 con 0x7fd1680feb50 2023-07-14T19:02:42.758 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:42.758 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:42.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd1540a11a0 msgr2=0x7fd1540a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:42.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd1540a11a0 0x7fd1540a3650 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fd158005ea0 tx=0x7fd158005e30 comp rx=0 tx=0).stop 2023-07-14T19:02:42.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 msgr2=0x7fd1680692c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:42.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 0x7fd1680692c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fd160004d10 tx=0x7fd160004df0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 shutdown_connections 2023-07-14T19:02:42.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd1540a11a0 0x7fd1540a3650 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd1681006d0 0x7fd16819ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd1680feb50 0x7fd1680692c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 --2- 172.21.15.78:0/1513207146 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd1680ffd40 0x7fd16819a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.755+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 >> 172.21.15.78:0/1513207146 conn(0x7fd1680fa150 msgr2=0x7fd168103130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:42.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.756+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 shutdown_connections 2023-07-14T19:02:42.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.756+0000 7fd1701a0700 1 -- 172.21.15.78:0/1513207146 wait complete. 2023-07-14T19:02:42.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:02:28.269050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:02:29.547771+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T19:02:32.797605+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T19:02:42.774 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:02:42.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 -- 172.21.15.78:0/1065838342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20801059a0 msgr2=0x7f2080107d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:42.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 --2- 172.21.15.78:0/1065838342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20801059a0 0x7f2080107d80 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f207800b0a0 tx=0x7f207800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 -- 172.21.15.78:0/1065838342 shutdown_connections 2023-07-14T19:02:42.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 --2- 172.21.15.78:0/1065838342 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2080100760 0x7f2080102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.901 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 --2- 172.21.15.78:0/1065838342 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f20801059a0 0x7f2080107d80 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 --2- 172.21.15.78:0/1065838342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2080103080 0x7f2080105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 -- 172.21.15.78:0/1065838342 >> 172.21.15.78:0/1065838342 conn(0x7f20800fa160 msgr2=0x7f20800fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:42.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 -- 172.21.15.78:0/1065838342 shutdown_connections 2023-07-14T19:02:42.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.895+0000 7f20873c7700 1 -- 172.21.15.78:0/1065838342 wait complete. 2023-07-14T19:02:42.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f20873c7700 1 Processor -- start 2023-07-14T19:02:42.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f20873c7700 1 -- start start 2023-07-14T19:02:42.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f20873c7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2080100760 0x7f20801000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:42.902 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f20873c7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2080103080 0x7f20800fe710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f20873c7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20801059a0 0x7f20800fec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f20873c7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20800ff3f0 con 0x7f2080103080 2023-07-14T19:02:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f20873c7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20800ff560 con 0x7f20801059a0 2023-07-14T19:02:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f20873c7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f208019fa30 con 0x7f2080100760 2023-07-14T19:02:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2084962700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2080103080 0x7f20800fe710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2084962700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2080103080 0x7f20800fe710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34692/0 (socket says 172.21.15.78:34692) 2023-07-14T19:02:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2084962700 1 -- 172.21.15.78:0/1152265616 learned_addr learned my addr 172.21.15.78:0/1152265616 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2085964700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20801059a0 0x7f20800fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2085163700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2080100760 0x7f20801000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2084962700 1 -- 172.21.15.78:0/1152265616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2080100760 msgr2=0x7f20801000d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2084962700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2080100760 0x7f20801000d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2084962700 1 -- 172.21.15.78:0/1152265616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20801059a0 msgr2=0x7f20800fec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2084962700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20801059a0 0x7f20800fec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2084962700 1 -- 172.21.15.78:0/1152265616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2078009d00 con 0x7f2080103080 2023-07-14T19:02:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.896+0000 7f2085163700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2080100760 0x7f20801000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.897+0000 7f2084962700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2080103080 0x7f20800fe710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f207c00c920 tx=0x7f207c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.897+0000 7f20767fc700 1 -- 172.21.15.78:0/1152265616 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c007b00 con 0x7f2080103080 2023-07-14T19:02:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.897+0000 7f20767fc700 1 -- 172.21.15.78:0/1152265616 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f207c00ce70 con 0x7f2080103080 2023-07-14T19:02:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.897+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f208019fd10 con 0x7f2080103080 2023-07-14T19:02:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.897+0000 7f20767fc700 1 -- 172.21.15.78:0/1152265616 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f207c005440 con 0x7f2080103080 2023-07-14T19:02:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.897+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20801a01a0 con 0x7f2080103080 2023-07-14T19:02:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.898+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20800fbcd0 con 0x7f2080103080 2023-07-14T19:02:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.899+0000 7f20767fc700 1 -- 172.21.15.78:0/1152265616 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f207c0055a0 con 0x7f2080103080 2023-07-14T19:02:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.899+0000 7f20767fc700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f206c0a1150 0x7f206c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.900+0000 7f20767fc700 1 -- 172.21.15.78:0/1152265616 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f207c0ccb70 con 0x7f2080103080 2023-07-14T19:02:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.900+0000 7f2085163700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f206c0a1150 0x7f206c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.900+0000 7f2085163700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f206c0a1150 0x7f206c0a3600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2070005fd0 tx=0x7f2070005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:42.903+0000 7f20767fc700 1 -- 172.21.15.78:0/1152265616 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f207c096070 con 0x7f2080103080 2023-07-14T19:02:43.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.091+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f208002ca70 con 0x7f2080103080 2023-07-14T19:02:43.095 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.091+0000 7f20767fc700 1 -- 172.21.15.78:0/1152265616 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7f207c01d020 con 0x7f2080103080 2023-07-14T19:02:43.096 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:02:43.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f206c0a1150 msgr2=0x7f206c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f206c0a1150 0x7f206c0a3600 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2070005fd0 tx=0x7f2070005ee0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2080103080 msgr2=0x7f20800fe710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2080103080 0x7f20800fe710 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f207c00c920 tx=0x7f207c00cc30 comp rx=0 tx=0).stop 2023-07-14T19:02:43.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 shutdown_connections 2023-07-14T19:02:43.098 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f206c0a1150 0x7f206c0a3600 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f20801059a0 0x7f20800fec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2080100760 0x7f20801000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 --2- 172.21.15.78:0/1152265616 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2080103080 0x7f20800fe710 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 >> 172.21.15.78:0/1152265616 conn(0x7f20800fa160 msgr2=0x7f20801013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:43.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 shutdown_connections 2023-07-14T19:02:43.099 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.094+0000 7f20873c7700 1 -- 172.21.15.78:0/1152265616 wait complete. 2023-07-14T19:02:43.109 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:02:43.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.229+0000 7f89e05f5700 1 -- 172.21.15.78:0/1266402449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 msgr2=0x7f89d8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.229+0000 7f89e05f5700 1 --2- 172.21.15.78:0/1266402449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 0x7f89d8106f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f89d000b0a0 tx=0x7f89d000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 -- 172.21.15.78:0/1266402449 shutdown_connections 2023-07-14T19:02:43.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 --2- 172.21.15.78:0/1266402449 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89d80feb10 0x7f89d80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 --2- 172.21.15.78:0/1266402449 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 0x7f89d8106f50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 --2- 172.21.15.78:0/1266402449 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89d80ffd00 0x7f89d8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 -- 172.21.15.78:0/1266402449 >> 172.21.15.78:0/1266402449 conn(0x7f89d80fa130 msgr2=0x7f89d80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:43.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 -- 172.21.15.78:0/1266402449 shutdown_connections 2023-07-14T19:02:43.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 -- 172.21.15.78:0/1266402449 wait complete. 2023-07-14T19:02:43.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 Processor -- start 2023-07-14T19:02:43.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 -- start start 2023-07-14T19:02:43.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89d80feb10 0x7f89d81947e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89d80ffd00 0x7f89d8194d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 0x7f89d819bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d819c360 con 0x7f89d8100690 2023-07-14T19:02:43.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d819c4a0 con 0x7f89d80ffd00 2023-07-14T19:02:43.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89e05f5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89d819c630 con 0x7f89d80feb10 2023-07-14T19:02:43.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.230+0000 7f89deb92700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 0x7f89d819bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89deb92700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 0x7f89d819bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34706/0 (socket says 172.21.15.78:34706) 2023-07-14T19:02:43.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89de391700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89d80feb10 0x7f89d81947e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89deb92700 1 -- 172.21.15.78:0/415678950 learned_addr learned my addr 172.21.15.78:0/415678950 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:43.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89ddb90700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89d80ffd00 0x7f89d8194d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89deb92700 1 -- 172.21.15.78:0/415678950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89d80feb10 msgr2=0x7f89d81947e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89deb92700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89d80feb10 0x7f89d81947e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89deb92700 1 -- 172.21.15.78:0/415678950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89d80ffd00 msgr2=0x7f89d8194d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89deb92700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89d80ffd00 0x7f89d8194d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89deb92700 1 -- 172.21.15.78:0/415678950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89d0009d00 con 0x7f89d8100690 2023-07-14T19:02:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89deb92700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 0x7f89d819bd90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f89d00062a0 tx=0x7f89d0004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89cf7fe700 1 -- 172.21.15.78:0/415678950 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d000e040 con 0x7f89d8100690 2023-07-14T19:02:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89cf7fe700 1 -- 172.21.15.78:0/415678950 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f89d001bcf0 con 0x7f89d8100690 2023-07-14T19:02:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89cf7fe700 1 -- 172.21.15.78:0/415678950 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89d0005950 con 0x7f89d8100690 2023-07-14T19:02:43.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89d819c8b0 con 0x7f89d8100690 2023-07-14T19:02:43.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.231+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89d819cd30 con 0x7f89d8100690 2023-07-14T19:02:43.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.232+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89d81059a0 con 0x7f89d8100690 2023-07-14T19:02:43.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.233+0000 7f89cf7fe700 1 -- 172.21.15.78:0/415678950 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f89d0005040 con 0x7f89d8100690 2023-07-14T19:02:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.234+0000 7f89cf7fe700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f89c40a11d0 0x7f89c40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.234+0000 7f89cf7fe700 1 -- 172.21.15.78:0/415678950 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f89d00cecb0 con 0x7f89d8100690 2023-07-14T19:02:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.234+0000 7f89de391700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f89c40a11d0 0x7f89c40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.234+0000 7f89de391700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f89c40a11d0 0x7f89c40a3680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f89d80ffb60 tx=0x7f89d400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:43.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.237+0000 7f89cf7fe700 1 -- 172.21.15.78:0/415678950 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f89d00982a0 con 0x7f89d8100690 2023-07-14T19:02:43.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.427+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89d802cb10 con 0x7f89d8100690 2023-07-14T19:02:43.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.428+0000 7f89cf7fe700 1 -- 172.21.15.78:0/415678950 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89d009b8c0 con 0x7f89d8100690 2023-07-14T19:02:43.432 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:02:43.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f89c40a11d0 msgr2=0x7f89c40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f89c40a11d0 0x7f89c40a3680 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f89d80ffb60 tx=0x7f89d400a3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 msgr2=0x7f89d819bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 0x7f89d819bd90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f89d00062a0 tx=0x7f89d0004ce0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 shutdown_connections 2023-07-14T19:02:43.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f89c40a11d0 0x7f89c40a3680 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f89d8100690 0x7f89d819bd90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f89d80feb10 0x7f89d81947e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 --2- 172.21.15.78:0/415678950 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f89d80ffd00 0x7f89d8194d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 >> 172.21.15.78:0/415678950 conn(0x7f89d80fa130 msgr2=0x7f89d80fc2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:43.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 shutdown_connections 2023-07-14T19:02:43.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.430+0000 7f89e05f5700 1 -- 172.21.15.78:0/415678950 wait complete. 2023-07-14T19:02:43.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:43 smithi078 ceph-mon[127425]: pgmap v261: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-07-14T19:02:43.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:43 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1513207146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:43.445 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:43 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1152265616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:43.446 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:02:43.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:43 smithi088 ceph-mon[128973]: pgmap v261: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-07-14T19:02:43.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:43 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1513207146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:43.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:43 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1152265616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:43.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 -- 172.21.15.78:0/714694510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940fed40 msgr2=0x7fde940ff190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 --2- 172.21.15.78:0/714694510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940fed40 0x7fde940ff190 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fde84009990 tx=0x7fde84009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 -- 172.21.15.78:0/714694510 shutdown_connections 2023-07-14T19:02:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 --2- 172.21.15.78:0/714694510 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde941000a0 0x7fde941004b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 --2- 172.21.15.78:0/714694510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde940ff6d0 0x7fde94106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 --2- 172.21.15.78:0/714694510 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940fed40 0x7fde940ff190 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 -- 172.21.15.78:0/714694510 >> 172.21.15.78:0/714694510 conn(0x7fde940f9860 msgr2=0x7fde940fbc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:43.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 -- 172.21.15.78:0/714694510 shutdown_connections 2023-07-14T19:02:43.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.567+0000 7fde98b5e700 1 -- 172.21.15.78:0/714694510 wait complete. 2023-07-14T19:02:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde98b5e700 1 Processor -- start 2023-07-14T19:02:43.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde98b5e700 1 -- start start 2023-07-14T19:02:43.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde98b5e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde940fed40 0x7fde94194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde98b5e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940ff6d0 0x7fde94194c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde98b5e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde941000a0 0x7fde9419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde98b5e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9419c280 con 0x7fde940ff6d0 2023-07-14T19:02:43.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde98b5e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9419c3c0 con 0x7fde941000a0 2023-07-14T19:02:43.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde98b5e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde9419c550 con 0x7fde940fed40 2023-07-14T19:02:43.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.568+0000 7fde91d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940ff6d0 0x7fde94194c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde91d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940ff6d0 0x7fde94194c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34718/0 (socket says 172.21.15.78:34718) 2023-07-14T19:02:43.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde91d9b700 1 -- 172.21.15.78:0/355390436 learned_addr learned my addr 172.21.15.78:0/355390436 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:43.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde9259c700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde940fed40 0x7fde94194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde92d9d700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde941000a0 0x7fde9419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde91d9b700 1 -- 172.21.15.78:0/355390436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde940fed40 msgr2=0x7fde94194700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde91d9b700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde940fed40 0x7fde94194700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.576 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde91d9b700 1 -- 172.21.15.78:0/355390436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde941000a0 msgr2=0x7fde9419bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde91d9b700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde941000a0 0x7fde9419bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde91d9b700 1 -- 172.21.15.78:0/355390436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde840095f0 con 0x7fde940ff6d0 2023-07-14T19:02:43.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde91d9b700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940ff6d0 0x7fde94194c40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fde840059d0 tx=0x7fde84004790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:43.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde837fe700 1 -- 172.21.15.78:0/355390436 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8401f070 con 0x7fde940ff6d0 2023-07-14T19:02:43.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde837fe700 1 -- 172.21.15.78:0/355390436 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fde84019600 con 0x7fde940ff6d0 2023-07-14T19:02:43.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde940fdc30 con 0x7fde940ff6d0 2023-07-14T19:02:43.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde837fe700 1 -- 172.21.15.78:0/355390436 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde8400f6a0 con 0x7fde940ff6d0 2023-07-14T19:02:43.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.569+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde940fe130 con 0x7fde940ff6d0 2023-07-14T19:02:43.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.570+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde940feb20 con 0x7fde940ff6d0 2023-07-14T19:02:43.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.571+0000 7fde837fe700 1 -- 172.21.15.78:0/355390436 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fde84019770 con 0x7fde940ff6d0 2023-07-14T19:02:43.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.572+0000 7fde837fe700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fde7c0a11d0 0x7fde7c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.572+0000 7fde9259c700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fde7c0a11d0 0x7fde7c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.572+0000 7fde837fe700 1 -- 172.21.15.78:0/355390436 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fde840d0180 con 0x7fde940ff6d0 2023-07-14T19:02:43.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.572+0000 7fde9259c700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fde7c0a11d0 0x7fde7c0a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde94067560 tx=0x7fde8800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:43.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.576+0000 7fde837fe700 1 -- 172.21.15.78:0/355390436 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fde84099770 con 0x7fde940ff6d0 2023-07-14T19:02:43.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:43 smithi102 ceph-mon[129280]: pgmap v261: 129 pgs: 129 active+clean; 579 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 3 op/s 2023-07-14T19:02:43.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:43 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1513207146' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:43.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:43 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1152265616' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:43.765 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.761+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fde94105a30 con 0x7fde940ff6d0 2023-07-14T19:02:43.766 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.762+0000 7fde837fe700 1 -- 172.21.15.78:0/355390436 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7fde8409cd90 con 0x7fde940ff6d0 2023-07-14T19:02:43.766 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-07-14T19:02:43.767 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fde7c0a11d0 msgr2=0x7fde7c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fde7c0a11d0 0x7fde7c0a3680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde94067560 tx=0x7fde8800a380 comp rx=0 tx=0).stop 2023-07-14T19:02:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940ff6d0 msgr2=0x7fde94194c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940ff6d0 0x7fde94194c40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fde840059d0 tx=0x7fde84004790 comp rx=0 tx=0).stop 2023-07-14T19:02:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 shutdown_connections 2023-07-14T19:02:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fde7c0a11d0 0x7fde7c0a3680 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.768 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fde941000a0 0x7fde9419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fde940fed40 0x7fde94194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 --2- 172.21.15.78:0/355390436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fde940ff6d0 0x7fde94194c40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 >> 172.21.15.78:0/355390436 conn(0x7fde940f9860 msgr2=0x7fde940fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 shutdown_connections 2023-07-14T19:02:43.769 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.764+0000 7fde98b5e700 1 -- 172.21.15.78:0/355390436 wait complete. 2023-07-14T19:02:43.779 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T19:02:43.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 -- 172.21.15.78:0/1577082307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24100e90 msgr2=0x7eff241012e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 --2- 172.21.15.78:0/1577082307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24100e90 0x7eff241012e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff14009990 tx=0x7eff14009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 -- 172.21.15.78:0/1577082307 shutdown_connections 2023-07-14T19:02:43.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 --2- 172.21.15.78:0/1577082307 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff240fdec0 0x7eff240fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 --2- 172.21.15.78:0/1577082307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff24101820 0x7eff241080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 --2- 172.21.15.78:0/1577082307 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24100e90 0x7eff241012e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 -- 172.21.15.78:0/1577082307 >> 172.21.15.78:0/1577082307 conn(0x7eff240f98a0 msgr2=0x7eff240fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:43.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 -- 172.21.15.78:0/1577082307 shutdown_connections 2023-07-14T19:02:43.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 -- 172.21.15.78:0/1577082307 wait complete. 2023-07-14T19:02:43.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 Processor -- start 2023-07-14T19:02:43.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.901+0000 7eff29b38700 1 -- start start 2023-07-14T19:02:43.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff29b38700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff240fdec0 0x7eff24100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff29b38700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff24100e90 0x7eff240fee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff29b38700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24101820 0x7eff240ff380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff29b38700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240ffb20 con 0x7eff24101820 2023-07-14T19:02:43.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff29b38700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff240ffc90 con 0x7eff24100e90 2023-07-14T19:02:43.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff29b38700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2419f920 con 0x7eff240fdec0 2023-07-14T19:02:43.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24101820 0x7eff240ff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24101820 0x7eff240ff380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34734/0 (socket says 172.21.15.78:34734) 2023-07-14T19:02:43.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 -- 172.21.15.78:0/3096682887 learned_addr learned my addr 172.21.15.78:0/3096682887 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:43.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff237fe700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff240fdec0 0x7eff24100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff22ffd700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff24100e90 0x7eff240fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 -- 172.21.15.78:0/3096682887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff240fdec0 msgr2=0x7eff24100800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff240fdec0 0x7eff24100800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 -- 172.21.15.78:0/3096682887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff24100e90 msgr2=0x7eff240fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:43.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff24100e90 0x7eff240fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:43.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 -- 172.21.15.78:0/3096682887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff140095f0 con 0x7eff24101820 2023-07-14T19:02:43.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff23fff700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24101820 0x7eff240ff380 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00b970 tx=0x7eff1c00bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:43.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff20ff9700 1 -- 172.21.15.78:0/3096682887 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c00c930 con 0x7eff24101820 2023-07-14T19:02:43.910 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff20ff9700 1 -- 172.21.15.78:0/3096682887 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7eff1c014440 con 0x7eff24101820 2023-07-14T19:02:43.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff2419fc00 con 0x7eff24101820 2023-07-14T19:02:43.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff20ff9700 1 -- 172.21.15.78:0/3096682887 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff1c0075d0 con 0x7eff24101820 2023-07-14T19:02:43.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.902+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff241a0110 con 0x7eff24101820 2023-07-14T19:02:43.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.903+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff24106c50 con 0x7eff24101820 2023-07-14T19:02:43.911 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.904+0000 7eff20ff9700 1 -- 172.21.15.78:0/3096682887 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7eff1c0145b0 con 0x7eff24101820 2023-07-14T19:02:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.905+0000 7eff20ff9700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7eff0c0a11d0 0x7eff0c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.905+0000 7eff237fe700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7eff0c0a11d0 0x7eff0c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.905+0000 7eff20ff9700 1 -- 172.21.15.78:0/3096682887 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7eff1c006e40 con 0x7eff24101820 2023-07-14T19:02:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.905+0000 7eff237fe700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7eff0c0a11d0 0x7eff0c0a3680 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff240675a0 tx=0x7eff18007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:43.908+0000 7eff20ff9700 1 -- 172.21.15.78:0/3096682887 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7eff1c096010 con 0x7eff24101820 2023-07-14T19:02:44.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.103+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7eff2402cd70 con 0x7eff24101820 2023-07-14T19:02:44.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.104+0000 7eff20ff9700 1 -- 172.21.15.78:0/3096682887 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7eff1c019020 con 0x7eff24101820 2023-07-14T19:02:44.108 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:02:44.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7eff0c0a11d0 msgr2=0x7eff0c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7eff0c0a11d0 0x7eff0c0a3680 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7eff240675a0 tx=0x7eff18007400 comp rx=0 tx=0).stop 2023-07-14T19:02:44.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24101820 msgr2=0x7eff240ff380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24101820 0x7eff240ff380 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7eff1c00b970 tx=0x7eff1c00bc80 comp rx=0 tx=0).stop 2023-07-14T19:02:44.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 shutdown_connections 2023-07-14T19:02:44.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7eff0c0a11d0 0x7eff0c0a3680 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7eff24101820 0x7eff240ff380 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7eff240fdec0 0x7eff24100800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 --2- 172.21.15.78:0/3096682887 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7eff24100e90 0x7eff240fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 >> 172.21.15.78:0/3096682887 conn(0x7eff240f98a0 msgr2=0x7eff240fbbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:44.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 shutdown_connections 2023-07-14T19:02:44.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.106+0000 7eff29b38700 1 -- 172.21.15.78:0/3096682887 wait complete. 2023-07-14T19:02:44.122 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T19:02:44.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1107605428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 msgr2=0x7fd3b41081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1107605428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 0x7fd3b41081a0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00b0a0 tx=0x7fd3ac00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1107605428 shutdown_connections 2023-07-14T19:02:44.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1107605428 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 0x7fd3b41081a0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1107605428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3b40fdee0 0x7fd3b40fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1107605428 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3b4100f50 0x7fd3b41013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1107605428 >> 172.21.15.78:0/1107605428 conn(0x7fd3b40f98a0 msgr2=0x7fd3b40fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:44.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1107605428 shutdown_connections 2023-07-14T19:02:44.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.241+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1107605428 wait complete. 2023-07-14T19:02:44.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3ba4d8700 1 Processor -- start 2023-07-14T19:02:44.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3ba4d8700 1 -- start start 2023-07-14T19:02:44.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3ba4d8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3b40fdee0 0x7fd3b4195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3ba4d8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3b4100f50 0x7fd3b4195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3ba4d8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 0x7fd3b419cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3ba4d8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b419d500 con 0x7fd3b41018e0 2023-07-14T19:02:44.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3ba4d8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b419d640 con 0x7fd3b40fdee0 2023-07-14T19:02:44.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3ba4d8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3b419d7d0 con 0x7fd3b4100f50 2023-07-14T19:02:44.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3b8a75700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 0x7fd3b419cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3b8a75700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 0x7fd3b419cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34746/0 (socket says 172.21.15.78:34746) 2023-07-14T19:02:44.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3b8a75700 1 -- 172.21.15.78:0/1573792959 learned_addr learned my addr 172.21.15.78:0/1573792959 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:44.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3b3fff700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3b40fdee0 0x7fd3b4195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.242+0000 7fd3b37fe700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3b4100f50 0x7fd3b4195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b8a75700 1 -- 172.21.15.78:0/1573792959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3b4100f50 msgr2=0x7fd3b4195ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b8a75700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3b4100f50 0x7fd3b4195ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b8a75700 1 -- 172.21.15.78:0/1573792959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3b40fdee0 msgr2=0x7fd3b4195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b8a75700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3b40fdee0 0x7fd3b4195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b8a75700 1 -- 172.21.15.78:0/1573792959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3ac009d00 con 0x7fd3b41018e0 2023-07-14T19:02:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b37fe700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3b4100f50 0x7fd3b4195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b8a75700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 0x7fd3b419cf30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00b780 tx=0x7fd3ac00bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:44.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b17fa700 1 -- 172.21.15.78:0/1573792959 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac00e040 con 0x7fd3b41018e0 2023-07-14T19:02:44.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b17fa700 1 -- 172.21.15.78:0/1573792959 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd3ac004e60 con 0x7fd3b41018e0 2023-07-14T19:02:44.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3b17fa700 1 -- 172.21.15.78:0/1573792959 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3ac0052a0 con 0x7fd3b41018e0 2023-07-14T19:02:44.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3b419da50 con 0x7fd3b41018e0 2023-07-14T19:02:44.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.243+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3b419ded0 con 0x7fd3b41018e0 2023-07-14T19:02:44.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.244+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3b410b230 con 0x7fd3b41018e0 2023-07-14T19:02:44.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.245+0000 7fd3b17fa700 1 -- 172.21.15.78:0/1573792959 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd3ac01d470 con 0x7fd3b41018e0 2023-07-14T19:02:44.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.245+0000 7fd3b17fa700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd39c0a11d0 0x7fd39c0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.246+0000 7fd3b17fa700 1 -- 172.21.15.78:0/1573792959 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd3ac0ce8b0 con 0x7fd3b41018e0 2023-07-14T19:02:44.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.246+0000 7fd3b3fff700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd39c0a11d0 0x7fd39c0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.246+0000 7fd3b3fff700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd39c0a11d0 0x7fd39c0a3680 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40674e0 tx=0x7fd3a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:44.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.249+0000 7fd3b17fa700 1 -- 172.21.15.78:0/1573792959 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd3ac097e40 con 0x7fd3b41018e0 2023-07-14T19:02:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.451+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd3b410b510 con 0x7fd3b41018e0 2023-07-14T19:02:44.455 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.451+0000 7fd3b17fa700 1 -- 172.21.15.78:0/1573792959 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+588 (secure 0 0 0) 0x7fd3b410b510 con 0x7fd3b41018e0 2023-07-14T19:02:44.456 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/378103227 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:44.456 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1800545696 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:44.456 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3249522727 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:44.456 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/378103227 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:44.456 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1797764467 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:44.456 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3249522727 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:44.456 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/21455120 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:44.457 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3838414629 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:44.457 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1459224377 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:44.457 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2696766201 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:44.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd39c0a11d0 msgr2=0x7fd39c0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd39c0a11d0 0x7fd39c0a3680 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd3b40674e0 tx=0x7fd3a8009260 comp rx=0 tx=0).stop 2023-07-14T19:02:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 msgr2=0x7fd3b419cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 0x7fd3b419cf30 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac00b780 tx=0x7fd3ac00bd40 comp rx=0 tx=0).stop 2023-07-14T19:02:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 shutdown_connections 2023-07-14T19:02:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd39c0a11d0 0x7fd39c0a3680 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd3b40fdee0 0x7fd3b4195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd3b41018e0 0x7fd3b419cf30 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 --2- 172.21.15.78:0/1573792959 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd3b4100f50 0x7fd3b4195ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 >> 172.21.15.78:0/1573792959 conn(0x7fd3b40f98a0 msgr2=0x7fd3b40fbb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.453+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 shutdown_connections 2023-07-14T19:02:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.454+0000 7fd3ba4d8700 1 -- 172.21.15.78:0/1573792959 wait complete. 2023-07-14T19:02:44.459 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2023-07-14T19:02:44.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:44 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/415678950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:44 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/355390436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:44 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3096682887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:44 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/415678950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:44 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/355390436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.468 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:44 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3096682887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.469 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T19:02:44.469 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:02:44.487 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:44.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:44 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/415678950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:44 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/355390436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:44 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3096682887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:02:44.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 -- 172.21.15.78:0/121124215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac105920 msgr2=0x7fe4ac107d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 --2- 172.21.15.78:0/121124215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac105920 0x7fe4ac107d20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fe4a400b0a0 tx=0x7fe4a400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 -- 172.21.15.78:0/121124215 shutdown_connections 2023-07-14T19:02:44.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 --2- 172.21.15.78:0/121124215 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac105920 0x7fe4ac107d20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 --2- 172.21.15.78:0/121124215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac1006e0 0x7fe4ac102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 --2- 172.21.15.78:0/121124215 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4ac103000 0x7fe4ac1053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 -- 172.21.15.78:0/121124215 >> 172.21.15.78:0/121124215 conn(0x7fe4ac0fa100 msgr2=0x7fe4ac0fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:44.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 -- 172.21.15.78:0/121124215 shutdown_connections 2023-07-14T19:02:44.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 -- 172.21.15.78:0/121124215 wait complete. 2023-07-14T19:02:44.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 Processor -- start 2023-07-14T19:02:44.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.645+0000 7fe4b2507700 1 -- start start 2023-07-14T19:02:44.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b2507700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4ac1006e0 0x7fe4ac195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b2507700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac103000 0x7fe4ac195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b2507700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac105920 0x7fe4ac19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b2507700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac19d3f0 con 0x7fe4ac103000 2023-07-14T19:02:44.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b2507700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac19d530 con 0x7fe4ac1006e0 2023-07-14T19:02:44.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b2507700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4ac19d6c0 con 0x7fe4ac105920 2023-07-14T19:02:44.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac103000 0x7fe4ac195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac103000 0x7fe4ac195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34756/0 (socket says 172.21.15.78:34756) 2023-07-14T19:02:44.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 -- 172.21.15.78:0/4272386154 learned_addr learned my addr 172.21.15.78:0/4272386154 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:44.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b0aa4700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac105920 0x7fe4ac19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4abfff700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4ac1006e0 0x7fe4ac195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 -- 172.21.15.78:0/4272386154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac105920 msgr2=0x7fe4ac19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac105920 0x7fe4ac19ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 -- 172.21.15.78:0/4272386154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4ac1006e0 msgr2=0x7fe4ac195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4ac1006e0 0x7fe4ac195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 -- 172.21.15.78:0/4272386154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a4009d00 con 0x7fe4ac103000 2023-07-14T19:02:44.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4ab7fe700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac103000 0x7fe4ac195db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00c920 tx=0x7fe49c00cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:44.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4a97fa700 1 -- 172.21.15.78:0/4272386154 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c007b00 con 0x7fe4ac103000 2023-07-14T19:02:44.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4ac19d9a0 con 0x7fe4ac103000 2023-07-14T19:02:44.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4a97fa700 1 -- 172.21.15.78:0/4272386154 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe49c004cc0 con 0x7fe4ac103000 2023-07-14T19:02:44.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4a97fa700 1 -- 172.21.15.78:0/4272386154 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe49c005490 con 0x7fe4ac103000 2023-07-14T19:02:44.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.646+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4ac19df00 con 0x7fe4ac103000 2023-07-14T19:02:44.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.647+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4ac0fbc70 con 0x7fe4ac103000 2023-07-14T19:02:44.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.648+0000 7fe4a97fa700 1 -- 172.21.15.78:0/4272386154 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe49c004750 con 0x7fe4ac103000 2023-07-14T19:02:44.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.649+0000 7fe4a97fa700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1220 0x7fe4940a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.649+0000 7fe4a97fa700 1 -- 172.21.15.78:0/4272386154 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe49c0ccbb0 con 0x7fe4ac103000 2023-07-14T19:02:44.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.649+0000 7fe4abfff700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1220 0x7fe4940a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.649+0000 7fe4abfff700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1220 0x7fe4940a36d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0005ea0 tx=0x7fe4a0005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:44.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.652+0000 7fe4a97fa700 1 -- 172.21.15.78:0/4272386154 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe49c0960b0 con 0x7fe4ac103000 2023-07-14T19:02:44.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.842+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4ac02c9d0 con 0x7fe4ac103000 2023-07-14T19:02:44.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.844+0000 7fe4a97fa700 1 -- 172.21.15.78:0/4272386154 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7fe49c01d020 con 0x7fe4ac103000 2023-07-14T19:02:44.848 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:44.848 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1220 msgr2=0x7fe4940a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1220 0x7fe4940a36d0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0005ea0 tx=0x7fe4a0005e30 comp rx=0 tx=0).stop 2023-07-14T19:02:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac103000 msgr2=0x7fe4ac195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac103000 0x7fe4ac195db0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fe49c00c920 tx=0x7fe49c00cce0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 shutdown_connections 2023-07-14T19:02:44.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe4940a1220 0x7fe4940a36d0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe4ac1006e0 0x7fe4ac195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe4ac105920 0x7fe4ac19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 --2- 172.21.15.78:0/4272386154 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe4ac103000 0x7fe4ac195db0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 >> 172.21.15.78:0/4272386154 conn(0x7fe4ac0fa100 msgr2=0x7fe4ac101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 shutdown_connections 2023-07-14T19:02:44.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.846+0000 7fe4b2507700 1 -- 172.21.15.78:0/4272386154 wait complete. 2023-07-14T19:02:44.862 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:02:28.269050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:02:29.547771+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T19:02:32.797605+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T19:02:44.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:02:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/3811206373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40feb10 msgr2=0x7fd8b40fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/3811206373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40feb10 0x7fd8b40fef20 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd8a4009990 tx=0x7fd8a4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/3811206373 shutdown_connections 2023-07-14T19:02:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/3811206373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4100690 0x7fd8b4106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/3811206373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40feb10 0x7fd8b40fef20 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/3811206373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8b40ffd00 0x7fd8b4100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/3811206373 >> 172.21.15.78:0/3811206373 conn(0x7fd8b40fa130 msgr2=0x7fd8b40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/3811206373 shutdown_connections 2023-07-14T19:02:44.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/3811206373 wait complete. 2023-07-14T19:02:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 Processor -- start 2023-07-14T19:02:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.983+0000 7fd8bb7ef700 1 -- start start 2023-07-14T19:02:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8bb7ef700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8b40feb10 0x7fd8b4069360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8bb7ef700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40ffd00 0x7fd8b419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8bb7ef700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4100690 0x7fd8b419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8bb7ef700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b419d410 con 0x7fd8b40ffd00 2023-07-14T19:02:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8bb7ef700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b419d580 con 0x7fd8b4100690 2023-07-14T19:02:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8bb7ef700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b419d740 con 0x7fd8b40feb10 2023-07-14T19:02:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40ffd00 0x7fd8b419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40ffd00 0x7fd8b419a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34770/0 (socket says 172.21.15.78:34770) 2023-07-14T19:02:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 -- 172.21.15.78:0/2958344296 learned_addr learned my addr 172.21.15.78:0/2958344296 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b958b700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8b40feb10 0x7fd8b4069360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 -- 172.21.15.78:0/2958344296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8b40feb10 msgr2=0x7fd8b4069360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8b40feb10 0x7fd8b4069360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 -- 172.21.15.78:0/2958344296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4100690 msgr2=0x7fd8b419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:44.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4100690 0x7fd8b419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:44.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 -- 172.21.15.78:0/2958344296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a40095f0 con 0x7fd8b40ffd00 2023-07-14T19:02:44.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8b8d8a700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40ffd00 0x7fd8b419a5f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000c9b0 tx=0x7fd8b000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:44.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8aa7fc700 1 -- 172.21.15.78:0/2958344296 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0007b00 con 0x7fd8b40ffd00 2023-07-14T19:02:44.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b419da20 con 0x7fd8b40ffd00 2023-07-14T19:02:44.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8aa7fc700 1 -- 172.21.15.78:0/2958344296 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd8b0007c60 con 0x7fd8b40ffd00 2023-07-14T19:02:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8aa7fc700 1 -- 172.21.15.78:0/2958344296 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8b0018660 con 0x7fd8b40ffd00 2023-07-14T19:02:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.984+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b419dfb0 con 0x7fd8b40ffd00 2023-07-14T19:02:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.985+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b41060f0 con 0x7fd8b40ffd00 2023-07-14T19:02:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.986+0000 7fd8aa7fc700 1 -- 172.21.15.78:0/2958344296 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd8b000f450 con 0x7fd8b40ffd00 2023-07-14T19:02:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.987+0000 7fd8aa7fc700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8a00a1150 0x7fd8a00a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.987+0000 7fd8b958b700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8a00a1150 0x7fd8a00a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.987+0000 7fd8aa7fc700 1 -- 172.21.15.78:0/2958344296 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd8b00ccd40 con 0x7fd8b40ffd00 2023-07-14T19:02:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.987+0000 7fd8b958b700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8a00a1150 0x7fd8a00a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400c010 tx=0x7fd8a4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:44.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:44.990+0000 7fd8aa7fc700 1 -- 172.21.15.78:0/2958344296 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd8b0096330 con 0x7fd8b40ffd00 2023-07-14T19:02:45.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.181+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd8b402ca70 con 0x7fd8b40ffd00 2023-07-14T19:02:45.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.181+0000 7fd8aa7fc700 1 -- 172.21.15.78:0/2958344296 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd8b000fe60 con 0x7fd8b40ffd00 2023-07-14T19:02:45.186 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:02:45.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8a00a1150 msgr2=0x7fd8a00a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8a00a1150 0x7fd8a00a3600 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fd8a400c010 tx=0x7fd8a4005fd0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40ffd00 msgr2=0x7fd8b419a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40ffd00 0x7fd8b419a5f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fd8b000c9b0 tx=0x7fd8b000ccc0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 shutdown_connections 2023-07-14T19:02:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8a00a1150 0x7fd8a00a3600 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd8b4100690 0x7fd8b419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8b40feb10 0x7fd8b4069360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.183+0000 7fd8bb7ef700 1 --2- 172.21.15.78:0/2958344296 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8b40ffd00 0x7fd8b419a5f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.184+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 >> 172.21.15.78:0/2958344296 conn(0x7fd8b40fa130 msgr2=0x7fd8b4103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.184+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 shutdown_connections 2023-07-14T19:02:45.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.184+0000 7fd8bb7ef700 1 -- 172.21.15.78:0/2958344296 wait complete. 2023-07-14T19:02:45.199 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:02:45.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 -- 172.21.15.78:0/80855977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640ffd00 msgr2=0x7fe764100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 --2- 172.21.15.78:0/80855977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640ffd00 0x7fe764100150 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe754009990 tx=0x7fe754009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 -- 172.21.15.78:0/80855977 shutdown_connections 2023-07-14T19:02:45.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 --2- 172.21.15.78:0/80855977 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe7640feb10 0x7fe7640fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 --2- 172.21.15.78:0/80855977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe764100690 0x7fe764106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 --2- 172.21.15.78:0/80855977 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640ffd00 0x7fe764100150 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 -- 172.21.15.78:0/80855977 >> 172.21.15.78:0/80855977 conn(0x7fe7640fa130 msgr2=0x7fe7640fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:45.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 -- 172.21.15.78:0/80855977 shutdown_connections 2023-07-14T19:02:45.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 -- 172.21.15.78:0/80855977 wait complete. 2023-07-14T19:02:45.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 Processor -- start 2023-07-14T19:02:45.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 -- start start 2023-07-14T19:02:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.320+0000 7fe76b805700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640feb10 0x7fe764068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:45.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe76b805700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7640ffd00 0x7fe76419a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe76b805700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe764100690 0x7fe76419cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe76b805700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76419d410 con 0x7fe7640feb10 2023-07-14T19:02:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe76b805700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76419d580 con 0x7fe764100690 2023-07-14T19:02:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe76b805700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76419d740 con 0x7fe7640ffd00 2023-07-14T19:02:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640feb10 0x7fe764068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640feb10 0x7fe764068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34782/0 (socket says 172.21.15.78:34782) 2023-07-14T19:02:45.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 -- 172.21.15.78:0/40474191 learned_addr learned my addr 172.21.15.78:0/40474191 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe768da0700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7640ffd00 0x7fe76419a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe769da2700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe764100690 0x7fe76419cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 -- 172.21.15.78:0/40474191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7640ffd00 msgr2=0x7fe76419a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7640ffd00 0x7fe76419a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 -- 172.21.15.78:0/40474191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe764100690 msgr2=0x7fe76419cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe764100690 0x7fe76419cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 -- 172.21.15.78:0/40474191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7540095f0 con 0x7fe7640feb10 2023-07-14T19:02:45.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe768da0700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7640ffd00 0x7fe76419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:45.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe7695a1700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640feb10 0x7fe764068330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe76000d980 tx=0x7fe76000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:45.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe75a7fc700 1 -- 172.21.15.78:0/40474191 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76000de60 con 0x7fe7640feb10 2023-07-14T19:02:45.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe75a7fc700 1 -- 172.21.15.78:0/40474191 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe7600044b0 con 0x7fe7640feb10 2023-07-14T19:02:45.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76419da20 con 0x7fe7640feb10 2023-07-14T19:02:45.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe75a7fc700 1 -- 172.21.15.78:0/40474191 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe760003b40 con 0x7fe7640feb10 2023-07-14T19:02:45.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.321+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76419df30 con 0x7fe7640feb10 2023-07-14T19:02:45.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.322+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe76410a190 con 0x7fe7640feb10 2023-07-14T19:02:45.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.323+0000 7fe75a7fc700 1 -- 172.21.15.78:0/40474191 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe760004870 con 0x7fe7640feb10 2023-07-14T19:02:45.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.324+0000 7fe75a7fc700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe7500a11d0 0x7fe7500a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:45.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.324+0000 7fe768da0700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe7500a11d0 0x7fe7500a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.324+0000 7fe75a7fc700 1 -- 172.21.15.78:0/40474191 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe7600ccc10 con 0x7fe7640feb10 2023-07-14T19:02:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.324+0000 7fe768da0700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe7500a11d0 0x7fe7500a3680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe75400c010 tx=0x7fe75400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:45.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.327+0000 7fe75a7fc700 1 -- 172.21.15.78:0/40474191 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe760096110 con 0x7fe7640feb10 2023-07-14T19:02:45.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:45 smithi088 ceph-mon[128973]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2023-07-14T19:02:45.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:45 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1573792959' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:02:45.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:45 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/4272386154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:45.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.517+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe764105b10 con 0x7fe7640feb10 2023-07-14T19:02:45.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.517+0000 7fe75a7fc700 1 -- 172.21.15.78:0/40474191 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe760009940 con 0x7fe7640feb10 2023-07-14T19:02:45.521 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:02:45.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe7500a11d0 msgr2=0x7fe7500a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe7500a11d0 0x7fe7500a3680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fe75400c010 tx=0x7fe75400b540 comp rx=0 tx=0).stop 2023-07-14T19:02:45.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640feb10 msgr2=0x7fe764068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640feb10 0x7fe764068330 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fe76000d980 tx=0x7fe76000dd40 comp rx=0 tx=0).stop 2023-07-14T19:02:45.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 shutdown_connections 2023-07-14T19:02:45.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe7500a11d0 0x7fe7500a3680 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe764100690 0x7fe76419cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe7640feb10 0x7fe764068330 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 --2- 172.21.15.78:0/40474191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe7640ffd00 0x7fe76419a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 >> 172.21.15.78:0/40474191 conn(0x7fe7640fa130 msgr2=0x7fe7640fc460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:45.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 shutdown_connections 2023-07-14T19:02:45.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.519+0000 7fe76b805700 1 -- 172.21.15.78:0/40474191 wait complete. 2023-07-14T19:02:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:45 smithi078 ceph-mon[127425]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2023-07-14T19:02:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:45 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1573792959' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:02:45.534 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:45 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/4272386154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:45.535 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:02:45.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:45 smithi102 ceph-mon[129280]: pgmap v262: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 3 op/s 2023-07-14T19:02:45.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:45 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1573792959' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:02:45.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:45 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/4272386154' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:45.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/4055897565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00ffd00 msgr2=0x7ff6a0100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/4055897565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00ffd00 0x7ff6a0100150 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ff690009990 tx=0x7ff690009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/4055897565 shutdown_connections 2023-07-14T19:02:45.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/4055897565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6a00feb10 0x7ff6a00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/4055897565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a0100690 0x7ff6a0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/4055897565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00ffd00 0x7ff6a0100150 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/4055897565 >> 172.21.15.78:0/4055897565 conn(0x7ff6a00fa130 msgr2=0x7ff6a00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/4055897565 shutdown_connections 2023-07-14T19:02:45.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.656+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/4055897565 wait complete. 2023-07-14T19:02:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff6a4bfa700 1 Processor -- start 2023-07-14T19:02:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff6a4bfa700 1 -- start start 2023-07-14T19:02:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff6a4bfa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00feb10 0x7ff6a0068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff6a4bfa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a00ffd00 0x7ff6a019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff6a4bfa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6a0100690 0x7ff6a019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff6a4bfa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a019d380 con 0x7ff6a00feb10 2023-07-14T19:02:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff6a4bfa700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a019d4f0 con 0x7ff6a0100690 2023-07-14T19:02:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff6a4bfa700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6a019d6b0 con 0x7ff6a00ffd00 2023-07-14T19:02:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00feb10 0x7ff6a0068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00feb10 0x7ff6a0068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34794/0 (socket says 172.21.15.78:34794) 2023-07-14T19:02:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 -- 172.21.15.78:0/268563784 learned_addr learned my addr 172.21.15.78:0/268563784 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69ed9d700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6a0100690 0x7ff6a019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69dd9b700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a00ffd00 0x7ff6a019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 -- 172.21.15.78:0/268563784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a00ffd00 msgr2=0x7ff6a019a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a00ffd00 0x7ff6a019a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 -- 172.21.15.78:0/268563784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6a0100690 msgr2=0x7ff6a019cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6a0100690 0x7ff6a019cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 -- 172.21.15.78:0/268563784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6900095f0 con 0x7ff6a00feb10 2023-07-14T19:02:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69dd9b700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a00ffd00 0x7ff6a019a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.657+0000 7ff69e59c700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00feb10 0x7ff6a0068330 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff69400d980 tx=0x7ff69400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.658+0000 7ff68f7fe700 1 -- 172.21.15.78:0/268563784 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff694004020 con 0x7ff6a00feb10 2023-07-14T19:02:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.658+0000 7ff68f7fe700 1 -- 172.21.15.78:0/268563784 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff694009730 con 0x7ff6a00feb10 2023-07-14T19:02:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.658+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6a019d990 con 0x7ff6a00feb10 2023-07-14T19:02:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.658+0000 7ff68f7fe700 1 -- 172.21.15.78:0/268563784 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6940039c0 con 0x7ff6a00feb10 2023-07-14T19:02:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.658+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6a019dea0 con 0x7ff6a00feb10 2023-07-14T19:02:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.658+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6a01060f0 con 0x7ff6a00feb10 2023-07-14T19:02:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.660+0000 7ff68f7fe700 1 -- 172.21.15.78:0/268563784 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff6940098a0 con 0x7ff6a00feb10 2023-07-14T19:02:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.660+0000 7ff68f7fe700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff6880a1150 0x7ff6880a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.660+0000 7ff69dd9b700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff6880a1150 0x7ff6880a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.660+0000 7ff68f7fe700 1 -- 172.21.15.78:0/268563784 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff6940ccb90 con 0x7ff6a00feb10 2023-07-14T19:02:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.661+0000 7ff69dd9b700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff6880a1150 0x7ff6880a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff690006010 tx=0x7ff69000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:45.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.664+0000 7ff68f7fe700 1 -- 172.21.15.78:0/268563784 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff6940960f0 con 0x7ff6a00feb10 2023-07-14T19:02:45.855 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.851+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff6a002ca70 con 0x7ff6a00feb10 2023-07-14T19:02:45.856 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.852+0000 7ff68f7fe700 1 -- 172.21.15.78:0/268563784 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff694099710 con 0x7ff6a00feb10 2023-07-14T19:02:45.856 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-07-14T19:02:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff6880a1150 msgr2=0x7ff6880a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff6880a1150 0x7ff6880a3600 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7ff690006010 tx=0x7ff69000b540 comp rx=0 tx=0).stop 2023-07-14T19:02:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00feb10 msgr2=0x7ff6a0068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.858 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00feb10 0x7ff6a0068330 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff69400d980 tx=0x7ff69400dc90 comp rx=0 tx=0).stop 2023-07-14T19:02:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 shutdown_connections 2023-07-14T19:02:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff6880a1150 0x7ff6880a3600 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff6a0100690 0x7ff6a019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff6a00feb10 0x7ff6a0068330 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 --2- 172.21.15.78:0/268563784 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff6a00ffd00 0x7ff6a019a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 >> 172.21.15.78:0/268563784 conn(0x7ff6a00fa130 msgr2=0x7ff6a0103230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 shutdown_connections 2023-07-14T19:02:45.859 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.854+0000 7ff6a4bfa700 1 -- 172.21.15.78:0/268563784 wait complete. 2023-07-14T19:02:45.870 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T19:02:45.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 -- 172.21.15.78:0/830140232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 msgr2=0x7f953c0ff290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:45.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 --2- 172.21.15.78:0/830140232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 0x7f953c0ff290 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f9538009990 tx=0x7f9538009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 -- 172.21.15.78:0/830140232 shutdown_connections 2023-07-14T19:02:45.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 --2- 172.21.15.78:0/830140232 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953c0ff7d0 0x7f953c107080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 --2- 172.21.15.78:0/830140232 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f953c100130 0x7f953c100540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 --2- 172.21.15.78:0/830140232 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 0x7f953c0ff290 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:45.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 -- 172.21.15.78:0/830140232 >> 172.21.15.78:0/830140232 conn(0x7f953c0f98a0 msgr2=0x7f953c0fbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:45.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 -- 172.21.15.78:0/830140232 shutdown_connections 2023-07-14T19:02:45.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.993+0000 7f95432c8700 1 -- 172.21.15.78:0/830140232 wait complete. 2023-07-14T19:02:45.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95432c8700 1 Processor -- start 2023-07-14T19:02:45.999 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95432c8700 1 -- start start 2023-07-14T19:02:46.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95432c8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 0x7f953c195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95432c8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953c0ff7d0 0x7f953c195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95432c8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f953c100130 0x7f953c19cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.001 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95432c8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c19d490 con 0x7f953c0fee20 2023-07-14T19:02:46.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95432c8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c19d5d0 con 0x7f953c0ff7d0 2023-07-14T19:02:46.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95432c8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f953c19d760 con 0x7f953c100130 2023-07-14T19:02:46.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 0x7f953c195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 0x7f953c195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34810/0 (socket says 172.21.15.78:34810) 2023-07-14T19:02:46.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 -- 172.21.15.78:0/334494400 learned_addr learned my addr 172.21.15.78:0/334494400 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:46.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9540863700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953c0ff7d0 0x7f953c195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 -- 172.21.15.78:0/334494400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f953c100130 msgr2=0x7f953c19cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f953c100130 0x7f953c19cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 -- 172.21.15.78:0/334494400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953c0ff7d0 msgr2=0x7f953c195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953c0ff7d0 0x7f953c195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 -- 172.21.15.78:0/334494400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95380095f0 con 0x7f953c0fee20 2023-07-14T19:02:46.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9540863700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953c0ff7d0 0x7f953c195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:46.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f9541064700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 0x7f953c195910 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f952c00d980 tx=0x7f952c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:46.003 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.994+0000 7f95327fc700 1 -- 172.21.15.78:0/334494400 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952c00bb60 con 0x7f953c0fee20 2023-07-14T19:02:46.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.995+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f953c19da40 con 0x7f953c0fee20 2023-07-14T19:02:46.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.995+0000 7f95327fc700 1 -- 172.21.15.78:0/334494400 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f952c007620 con 0x7f953c0fee20 2023-07-14T19:02:46.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.995+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f953c19dfa0 con 0x7f953c0fee20 2023-07-14T19:02:46.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.995+0000 7f95327fc700 1 -- 172.21.15.78:0/334494400 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f952c00f3c0 con 0x7f953c0fee20 2023-07-14T19:02:46.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.996+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9520005320 con 0x7f953c0fee20 2023-07-14T19:02:46.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.997+0000 7f95327fc700 1 -- 172.21.15.78:0/334494400 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f952c010460 con 0x7f953c0fee20 2023-07-14T19:02:46.004 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.997+0000 7f95327fc700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95280a1150 0x7f95280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.997+0000 7f9540863700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95280a1150 0x7f95280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.997+0000 7f95327fc700 1 -- 172.21.15.78:0/334494400 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f952c0cd1f0 con 0x7f953c0fee20 2023-07-14T19:02:46.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:45.998+0000 7f9540863700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95280a1150 0x7f95280a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f953800c010 tx=0x7f953800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:46.005 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.001+0000 7f95327fc700 1 -- 172.21.15.78:0/334494400 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f952c096750 con 0x7f953c0fee20 2023-07-14T19:02:46.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.191+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9520005f70 con 0x7f953c0fee20 2023-07-14T19:02:46.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.192+0000 7f95327fc700 1 -- 172.21.15.78:0/334494400 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) v1 ==== 103+0+11 (secure 0 0 0) 0x7f952c00de60 con 0x7f953c0fee20 2023-07-14T19:02:46.196 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:02:46.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95280a1150 msgr2=0x7f95280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95280a1150 0x7f95280a3600 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f953800c010 tx=0x7f953800b540 comp rx=0 tx=0).stop 2023-07-14T19:02:46.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 msgr2=0x7f953c195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 0x7f953c195910 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f952c00d980 tx=0x7f952c00dd40 comp rx=0 tx=0).stop 2023-07-14T19:02:46.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 shutdown_connections 2023-07-14T19:02:46.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95280a1150 0x7f95280a3600 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f953c100130 0x7f953c19cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f953c0fee20 0x7f953c195910 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 --2- 172.21.15.78:0/334494400 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f953c0ff7d0 0x7f953c195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 >> 172.21.15.78:0/334494400 conn(0x7f953c0f98a0 msgr2=0x7f953c103410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:46.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 shutdown_connections 2023-07-14T19:02:46.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.194+0000 7f95432c8700 1 -- 172.21.15.78:0/334494400 wait complete. 2023-07-14T19:02:46.209 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T19:02:46.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 -- 172.21.15.78:0/1855353327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a0100770 msgr2=0x7f42a0107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/1855353327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a0100770 0x7f42a0107010 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f429800b0a0 tx=0x7f429800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 -- 172.21.15.78:0/1855353327 shutdown_connections 2023-07-14T19:02:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/1855353327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a00feb30 0x7f42a00fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/1855353327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a0100770 0x7f42a0107010 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/1855353327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a00ffdc0 0x7f42a0100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 -- 172.21.15.78:0/1855353327 >> 172.21.15.78:0/1855353327 conn(0x7f42a00fa150 msgr2=0x7f42a00fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:46.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 -- 172.21.15.78:0/1855353327 shutdown_connections 2023-07-14T19:02:46.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.327+0000 7f42a6ba4700 1 -- 172.21.15.78:0/1855353327 wait complete. 2023-07-14T19:02:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a6ba4700 1 Processor -- start 2023-07-14T19:02:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a6ba4700 1 -- start start 2023-07-14T19:02:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a6ba4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a00feb30 0x7f42a0069520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a6ba4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a00ffdc0 0x7f42a019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a6ba4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0100770 0x7f42a019cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a6ba4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a019d420 con 0x7f42a00ffdc0 2023-07-14T19:02:46.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a6ba4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a019d590 con 0x7f42a0100770 2023-07-14T19:02:46.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a6ba4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42a019d750 con 0x7f42a00feb30 2023-07-14T19:02:46.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f429ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a00ffdc0 0x7f42a019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.337 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f429ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a00ffdc0 0x7f42a019a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34826/0 (socket says 172.21.15.78:34826) 2023-07-14T19:02:46.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f429ffff700 1 -- 172.21.15.78:0/2971543045 learned_addr learned my addr 172.21.15.78:0/2971543045 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:46.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a4940700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a00feb30 0x7f42a0069520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.328+0000 7f42a5141700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0100770 0x7f42a019cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429ffff700 1 -- 172.21.15.78:0/2971543045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a00feb30 msgr2=0x7f42a0069520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429ffff700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a00feb30 0x7f42a0069520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429ffff700 1 -- 172.21.15.78:0/2971543045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0100770 msgr2=0x7f42a019cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429ffff700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0100770 0x7f42a019cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429ffff700 1 -- 172.21.15.78:0/2971543045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4298009d00 con 0x7f42a00ffdc0 2023-07-14T19:02:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429ffff700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a00ffdc0 0x7f42a019a5f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f428c00c920 tx=0x7f428c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429dffb700 1 -- 172.21.15.78:0/2971543045 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c007b00 con 0x7f42a00ffdc0 2023-07-14T19:02:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429dffb700 1 -- 172.21.15.78:0/2971543045 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f428c00ce70 con 0x7f42a00ffdc0 2023-07-14T19:02:46.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f429dffb700 1 -- 172.21.15.78:0/2971543045 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f428c005440 con 0x7f42a00ffdc0 2023-07-14T19:02:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42a019da30 con 0x7f42a00ffdc0 2023-07-14T19:02:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.329+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42a019def0 con 0x7f42a00ffdc0 2023-07-14T19:02:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.330+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42a010a1b0 con 0x7f42a00ffdc0 2023-07-14T19:02:46.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.331+0000 7f429dffb700 1 -- 172.21.15.78:0/2971543045 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f428c0055a0 con 0x7f42a00ffdc0 2023-07-14T19:02:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.331+0000 7f429dffb700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42900a1150 0x7f42900a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.332+0000 7f429dffb700 1 -- 172.21.15.78:0/2971543045 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f428c0cc6d0 con 0x7f42a00ffdc0 2023-07-14T19:02:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.332+0000 7f42a4940700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42900a1150 0x7f42900a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.332+0000 7f42a4940700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42900a1150 0x7f42900a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f429400ac50 tx=0x7f429400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:46.341 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.336+0000 7f429dffb700 1 -- 172.21.15.78:0/2971543045 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f428c095c30 con 0x7f42a00ffdc0 2023-07-14T19:02:46.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:46 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2958344296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:46 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/40474191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:46 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/268563784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.541+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f42a019e6f0 con 0x7f42a00ffdc0 2023-07-14T19:02:46.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.541+0000 7f429dffb700 1 -- 172.21.15.78:0/2971543045 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) v1 ==== 81+0+588 (secure 0 0 0) 0x7f428c099250 con 0x7f42a00ffdc0 2023-07-14T19:02:46.545 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/378103227 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:46.545 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1800545696 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:46.545 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3249522727 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/378103227 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1797764467 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3249522727 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/21455120 2023-07-15T18:56:30.036843+0000 2023-07-14T19:02:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3838414629 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1459224377 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:46.546 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2696766201 2023-07-15T18:56:08.189962+0000 2023-07-14T19:02:46.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42900a1150 msgr2=0x7f42900a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42900a1150 0x7f42900a3600 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f429400ac50 tx=0x7f429400a380 comp rx=0 tx=0).stop 2023-07-14T19:02:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a00ffdc0 msgr2=0x7f42a019a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a00ffdc0 0x7f42a019a5f0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f428c00c920 tx=0x7f428c00cc30 comp rx=0 tx=0).stop 2023-07-14T19:02:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 shutdown_connections 2023-07-14T19:02:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42900a1150 0x7f42900a3600 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42a0100770 0x7f42a019cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42a00feb30 0x7f42a0069520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 --2- 172.21.15.78:0/2971543045 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42a00ffdc0 0x7f42a019a5f0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 >> 172.21.15.78:0/2971543045 conn(0x7f42a00fa150 msgr2=0x7f42a00fc360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:46.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 shutdown_connections 2023-07-14T19:02:46.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.543+0000 7f42a6ba4700 1 -- 172.21.15.78:0/2971543045 wait complete. 2023-07-14T19:02:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:46 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2958344296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:46 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/40474191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:46 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/268563784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.550 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2023-07-14T19:02:46.559 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2023-07-14T19:02:46.559 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:46.560 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2023-07-14T19:02:46.595 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi078.stderr:ceph-fuse[164624]: fuse finished with error 0 and tester_r 0 2023-07-14T19:02:46.596 INFO:teuthology.orchestra.run:waiting for 300 2023-07-14T19:02:46.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:46 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2958344296' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:46 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/40474191' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:46 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/268563784' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:02:46.649 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2023-07-14T19:02:46.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:02:46.650 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2023-07-14T19:02:46.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:46.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.828+0000 7f8594c0c700 1 -- 172.21.15.78:0/70378327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8590100e20 msgr2=0x7f8590101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.828+0000 7f8594c0c700 1 --2- 172.21.15.78:0/70378327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8590100e20 0x7f8590101270 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8584009990 tx=0x7f8584009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.829+0000 7f8594c0c700 1 -- 172.21.15.78:0/70378327 shutdown_connections 2023-07-14T19:02:46.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.829+0000 7f8594c0c700 1 --2- 172.21.15.78:0/70378327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85900ffc30 0x7f8590100040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.829+0000 7f8594c0c700 1 --2- 172.21.15.78:0/70378327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85901017b0 0x7f8590108070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.829+0000 7f8594c0c700 1 --2- 172.21.15.78:0/70378327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8590100e20 0x7f8590101270 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.833 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.829+0000 7f8594c0c700 1 -- 172.21.15.78:0/70378327 >> 172.21.15.78:0/70378327 conn(0x7f85900fb210 msgr2=0x7f85900fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:46.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.829+0000 7f8594c0c700 1 -- 172.21.15.78:0/70378327 shutdown_connections 2023-07-14T19:02:46.834 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.829+0000 7f8594c0c700 1 -- 172.21.15.78:0/70378327 wait complete. 2023-07-14T19:02:46.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f8594c0c700 1 Processor -- start 2023-07-14T19:02:46.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f8594c0c700 1 -- start start 2023-07-14T19:02:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f8594c0c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85900ffc30 0x7f8590195840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f8594c0c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8590100e20 0x7f8590195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f8594c0c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85901017b0 0x7f859019cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f8594c0c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859019d3c0 con 0x7f85901017b0 2023-07-14T19:02:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f8594c0c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859019d500 con 0x7f8590100e20 2023-07-14T19:02:46.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f8594c0c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f859019d690 con 0x7f85900ffc30 2023-07-14T19:02:46.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85901017b0 0x7f859019cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85901017b0 0x7f859019cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34844/0 (socket says 172.21.15.78:34844) 2023-07-14T19:02:46.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 -- 172.21.15.78:0/1487727473 learned_addr learned my addr 172.21.15.78:0/1487727473 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:46.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858dd9b700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8590100e20 0x7f8590195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858e59c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85900ffc30 0x7f8590195840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 -- 172.21.15.78:0/1487727473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85900ffc30 msgr2=0x7f8590195840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85900ffc30 0x7f8590195840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 -- 172.21.15.78:0/1487727473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8590100e20 msgr2=0x7f8590195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:46.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8590100e20 0x7f8590195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:46.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 -- 172.21.15.78:0/1487727473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85840095f0 con 0x7f85901017b0 2023-07-14T19:02:46.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.830+0000 7f858ed9d700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85901017b0 0x7f859019cdf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f858800ef30 tx=0x7f858800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:46.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.831+0000 7f857f7fe700 1 -- 172.21.15.78:0/1487727473 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8588018070 con 0x7f85901017b0 2023-07-14T19:02:46.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.831+0000 7f857f7fe700 1 -- 172.21.15.78:0/1487727473 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f858800f040 con 0x7f85901017b0 2023-07-14T19:02:46.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.831+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f859019d910 con 0x7f85901017b0 2023-07-14T19:02:46.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.831+0000 7f857f7fe700 1 -- 172.21.15.78:0/1487727473 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f858800cbc0 con 0x7f85901017b0 2023-07-14T19:02:46.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.831+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f859019de20 con 0x7f85901017b0 2023-07-14T19:02:46.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.831+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85901071f0 con 0x7f85901017b0 2023-07-14T19:02:46.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.833+0000 7f857f7fe700 1 -- 172.21.15.78:0/1487727473 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f858800cd40 con 0x7f85901017b0 2023-07-14T19:02:46.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.833+0000 7f857f7fe700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85780a11a0 0x7f85780a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:46.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.833+0000 7f858e59c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85780a11a0 0x7f85780a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:46.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.833+0000 7f857f7fe700 1 -- 172.21.15.78:0/1487727473 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f85880ccbc0 con 0x7f85901017b0 2023-07-14T19:02:46.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.834+0000 7f858e59c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85780a11a0 0x7f85780a3650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8590100c80 tx=0x7f858000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:46.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:46.837+0000 7f857f7fe700 1 -- 172.21.15.78:0/1487727473 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f85880961b0 con 0x7f85901017b0 2023-07-14T19:02:47.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.024+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f859002cb30 con 0x7f85901017b0 2023-07-14T19:02:47.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.026+0000 7f857f7fe700 1 -- 172.21.15.78:0/1487727473 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7f85880997d0 con 0x7f85901017b0 2023-07-14T19:02:47.030 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:47.030 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:47.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85780a11a0 msgr2=0x7f85780a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85780a11a0 0x7f85780a3650 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8590100c80 tx=0x7f858000a380 comp rx=0 tx=0).stop 2023-07-14T19:02:47.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85901017b0 msgr2=0x7f859019cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85901017b0 0x7f859019cdf0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f858800ef30 tx=0x7f858800c3a0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 shutdown_connections 2023-07-14T19:02:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85780a11a0 0x7f85780a3650 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85901017b0 0x7f859019cdf0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85900ffc30 0x7f8590195840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 --2- 172.21.15.78:0/1487727473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8590100e20 0x7f8590195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 >> 172.21.15.78:0/1487727473 conn(0x7f85900fb210 msgr2=0x7f8590104330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 shutdown_connections 2023-07-14T19:02:47.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.028+0000 7f8594c0c700 1 -- 172.21.15.78:0/1487727473 wait complete. 2023-07-14T19:02:47.044 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:47.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 -- 172.21.15.78:0/936317174 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed08100680 msgr2=0x7fed08106f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 --2- 172.21.15.78:0/936317174 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed08100680 0x7fed08106f40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fed0000b0a0 tx=0x7fed0000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 -- 172.21.15.78:0/936317174 shutdown_connections 2023-07-14T19:02:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 --2- 172.21.15.78:0/936317174 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed08100680 0x7fed08106f40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 --2- 172.21.15.78:0/936317174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed080feb00 0x7fed080fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 --2- 172.21.15.78:0/936317174 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed080ffcf0 0x7fed08100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 -- 172.21.15.78:0/936317174 >> 172.21.15.78:0/936317174 conn(0x7fed080fa100 msgr2=0x7fed080fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 -- 172.21.15.78:0/936317174 shutdown_connections 2023-07-14T19:02:47.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.161+0000 7fed0f91c700 1 -- 172.21.15.78:0/936317174 wait complete. 2023-07-14T19:02:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0f91c700 1 Processor -- start 2023-07-14T19:02:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0f91c700 1 -- start start 2023-07-14T19:02:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0f91c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed080feb00 0x7fed08195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:47.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0f91c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed080ffcf0 0x7fed08195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:47.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0f91c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed08100680 0x7fed0819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:47.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0f91c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0819d4a0 con 0x7fed080feb00 2023-07-14T19:02:47.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0f91c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0819d5e0 con 0x7fed080ffcf0 2023-07-14T19:02:47.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0f91c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0819d770 con 0x7fed08100680 2023-07-14T19:02:47.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0d6b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed080feb00 0x7fed08195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:47.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0d6b8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed080feb00 0x7fed08195920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34856/0 (socket says 172.21.15.78:34856) 2023-07-14T19:02:47.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0d6b8700 1 -- 172.21.15.78:0/866254233 learned_addr learned my addr 172.21.15.78:0/866254233 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:47.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0ceb7700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed080ffcf0 0x7fed08195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:47.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0deb9700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed08100680 0x7fed0819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:47.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0d6b8700 1 -- 172.21.15.78:0/866254233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed08100680 msgr2=0x7fed0819ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0d6b8700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed08100680 0x7fed0819ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0d6b8700 1 -- 172.21.15.78:0/866254233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed080ffcf0 msgr2=0x7fed08195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0d6b8700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed080ffcf0 0x7fed08195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.162+0000 7fed0d6b8700 1 -- 172.21.15.78:0/866254233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed00009d00 con 0x7fed080feb00 2023-07-14T19:02:47.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.163+0000 7fed0d6b8700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed080feb00 0x7fed08195920 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00d980 tx=0x7fecfc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:47.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.163+0000 7fecfa7fc700 1 -- 172.21.15.78:0/866254233 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc004020 con 0x7fed080feb00 2023-07-14T19:02:47.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.163+0000 7fecfa7fc700 1 -- 172.21.15.78:0/866254233 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fecfc00f690 con 0x7fed080feb00 2023-07-14T19:02:47.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.163+0000 7fecfa7fc700 1 -- 172.21.15.78:0/866254233 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecfc0039e0 con 0x7fed080feb00 2023-07-14T19:02:47.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.163+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0819da50 con 0x7fed080feb00 2023-07-14T19:02:47.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.163+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0819df10 con 0x7fed080feb00 2023-07-14T19:02:47.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.164+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0810a190 con 0x7fed080feb00 2023-07-14T19:02:47.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.165+0000 7fecfa7fc700 1 -- 172.21.15.78:0/866254233 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fecfc00f800 con 0x7fed080feb00 2023-07-14T19:02:47.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.166+0000 7fecfa7fc700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a11a0 0x7fecf40a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:47.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.166+0000 7fed0ceb7700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a11a0 0x7fecf40a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.166+0000 7fecfa7fc700 1 -- 172.21.15.78:0/866254233 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fecfc0cca40 con 0x7fed080feb00 2023-07-14T19:02:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.166+0000 7fed0ceb7700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a11a0 0x7fecf40a3650 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fed04009c60 tx=0x7fed04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:47.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.169+0000 7fecfa7fc700 1 -- 172.21.15.78:0/866254233 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fecfc095fa0 con 0x7fed080feb00 2023-07-14T19:02:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:47 smithi078 ceph-mon[127425]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 389 B/s wr, 2 op/s 2023-07-14T19:02:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:47 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/334494400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:02:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:47 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2971543045' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:02:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:47 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1487727473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:47.361 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.358+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed0819e580 con 0x7fed080feb00 2023-07-14T19:02:47.363 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.359+0000 7fecfa7fc700 1 -- 172.21.15.78:0/866254233 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7fecfc0995c0 con 0x7fed080feb00 2023-07-14T19:02:47.363 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:47.364 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a11a0 msgr2=0x7fecf40a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a11a0 0x7fecf40a3650 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fed04009c60 tx=0x7fed04009260 comp rx=0 tx=0).stop 2023-07-14T19:02:47.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed080feb00 msgr2=0x7fed08195920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed080feb00 0x7fed08195920 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fecfc00d980 tx=0x7fecfc00dc90 comp rx=0 tx=0).stop 2023-07-14T19:02:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 shutdown_connections 2023-07-14T19:02:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a11a0 0x7fecf40a3650 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed08100680 0x7fed0819ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed080feb00 0x7fed08195920 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 --2- 172.21.15.78:0/866254233 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed080ffcf0 0x7fed08195e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.361+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 >> 172.21.15.78:0/866254233 conn(0x7fed080fa100 msgr2=0x7fed08067880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.362+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 shutdown_connections 2023-07-14T19:02:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.362+0000 7fed0f91c700 1 -- 172.21.15.78:0/866254233 wait complete. 2023-07-14T19:02:47.377 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-07-14T19:02:47.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:47 smithi088 ceph-mon[128973]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 389 B/s wr, 2 op/s 2023-07-14T19:02:47.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:47 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/334494400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:02:47.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:47 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2971543045' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:02:47.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:47 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1487727473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:47.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- 172.21.15.78:0/4153672792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c101790 msgr2=0x7ff53c108050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 --2- 172.21.15.78:0/4153672792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c101790 0x7ff53c108050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ff53400b0a0 tx=0x7ff53400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- 172.21.15.78:0/4153672792 shutdown_connections 2023-07-14T19:02:47.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 --2- 172.21.15.78:0/4153672792 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c101790 0x7ff53c108050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 --2- 172.21.15.78:0/4153672792 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff53c0ffc10 0x7ff53c100020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 --2- 172.21.15.78:0/4153672792 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff53c100e00 0x7ff53c101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- 172.21.15.78:0/4153672792 >> 172.21.15.78:0/4153672792 conn(0x7ff53c0fb210 msgr2=0x7ff53c0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:47.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- 172.21.15.78:0/4153672792 shutdown_connections 2023-07-14T19:02:47.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- 172.21.15.78:0/4153672792 wait complete. 2023-07-14T19:02:47.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 Processor -- start 2023-07-14T19:02:47.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- start start 2023-07-14T19:02:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c0ffc10 0x7ff53c0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff53c100e00 0x7ff53c072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff53c101790 0x7ff53c072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:47.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c073710 con 0x7ff53c0ffc10 2023-07-14T19:02:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c073880 con 0x7ff53c101790 2023-07-14T19:02:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.498+0000 7ff543a49700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff53c19f9e0 con 0x7ff53c100e00 2023-07-14T19:02:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c0ffc10 0x7ff53c0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c0ffc10 0x7ff53c0743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34868/0 (socket says 172.21.15.78:34868) 2023-07-14T19:02:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 -- 172.21.15.78:0/1676524571 learned_addr learned my addr 172.21.15.78:0/1676524571 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff540fe4700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff53c100e00 0x7ff53c072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:47.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff541fe6700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff53c101790 0x7ff53c072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 -- 172.21.15.78:0/1676524571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff53c100e00 msgr2=0x7ff53c072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff53c100e00 0x7ff53c072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 -- 172.21.15.78:0/1676524571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff53c101790 msgr2=0x7ff53c072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff53c101790 0x7ff53c072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 -- 172.21.15.78:0/1676524571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff534009d00 con 0x7ff53c0ffc10 2023-07-14T19:02:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5417e5700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c0ffc10 0x7ff53c0743f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff53800d980 tx=0x7ff53800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5327fc700 1 -- 172.21.15.78:0/1676524571 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538004020 con 0x7ff53c0ffc10 2023-07-14T19:02:47.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff53c19fcc0 con 0x7ff53c0ffc10 2023-07-14T19:02:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5327fc700 1 -- 172.21.15.78:0/1676524571 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff538009730 con 0x7ff53c0ffc10 2023-07-14T19:02:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff5327fc700 1 -- 172.21.15.78:0/1676524571 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff538003a00 con 0x7ff53c0ffc10 2023-07-14T19:02:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.499+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff53c1a01d0 con 0x7ff53c0ffc10 2023-07-14T19:02:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.500+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff53c1071f0 con 0x7ff53c0ffc10 2023-07-14T19:02:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.502+0000 7ff5327fc700 1 -- 172.21.15.78:0/1676524571 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff5380098a0 con 0x7ff53c0ffc10 2023-07-14T19:02:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.502+0000 7ff5327fc700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff5280a1150 0x7ff5280a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.502+0000 7ff540fe4700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff5280a1150 0x7ff5280a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.502+0000 7ff5327fc700 1 -- 172.21.15.78:0/1676524571 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff5380ccb80 con 0x7ff53c0ffc10 2023-07-14T19:02:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.503+0000 7ff540fe4700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff5280a1150 0x7ff5280a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff52c009c60 tx=0x7ff52c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:47.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.506+0000 7ff5327fc700 1 -- 172.21.15.78:0/1676524571 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff5380960e0 con 0x7ff53c0ffc10 2023-07-14T19:02:47.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:47 smithi102 ceph-mon[129280]: pgmap v263: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 389 B/s wr, 2 op/s 2023-07-14T19:02:47.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:47 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/334494400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:02:47.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:47 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2971543045' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:02:47.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:47 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1487727473' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:47.741 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:47.738+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff53c02c8e0 con 0x7ff53c0ffc10 2023-07-14T19:02:48.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.180+0000 7ff5327fc700 1 -- 172.21.15.78:0/1676524571 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff538099700 con 0x7ff53c0ffc10 2023-07-14T19:02:48.187 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff5280a1150 msgr2=0x7ff5280a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff5280a1150 0x7ff5280a3600 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ff52c009c60 tx=0x7ff52c009260 comp rx=0 tx=0).stop 2023-07-14T19:02:48.188 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c0ffc10 msgr2=0x7ff53c0743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c0ffc10 0x7ff53c0743f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff53800d980 tx=0x7ff53800dc90 comp rx=0 tx=0).stop 2023-07-14T19:02:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 shutdown_connections 2023-07-14T19:02:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff5280a1150 0x7ff5280a3600 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff53c101790 0x7ff53c072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff53c0ffc10 0x7ff53c0743f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.189 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 --2- 172.21.15.78:0/1676524571 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff53c100e00 0x7ff53c072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 >> 172.21.15.78:0/1676524571 conn(0x7ff53c0fb210 msgr2=0x7ff53c104330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 shutdown_connections 2023-07-14T19:02:48.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.183+0000 7ff543a49700 1 -- 172.21.15.78:0/1676524571 wait complete. 2023-07-14T19:02:48.201 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2023-07-14T19:02:48.201 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2023-07-14T19:02:48.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 -- 172.21.15.78:0/2849396684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff860101820 msgr2=0x7ff8601080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:48.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2849396684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff860101820 0x7ff8601080e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ff85800b0a0 tx=0x7ff85800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 -- 172.21.15.78:0/2849396684 shutdown_connections 2023-07-14T19:02:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2849396684 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff860101820 0x7ff8601080e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2849396684 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8600fdec0 0x7ff8600fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2849396684 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff860100e90 0x7ff8601012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 -- 172.21.15.78:0/2849396684 >> 172.21.15.78:0/2849396684 conn(0x7ff8600f98a0 msgr2=0x7ff8600fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 -- 172.21.15.78:0/2849396684 shutdown_connections 2023-07-14T19:02:48.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.324+0000 7ff8663a1700 1 -- 172.21.15.78:0/2849396684 wait complete. 2023-07-14T19:02:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff8663a1700 1 Processor -- start 2023-07-14T19:02:48.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff8663a1700 1 -- start start 2023-07-14T19:02:48.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff8663a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8600fdec0 0x7ff8601959e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff8663a1700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff860100e90 0x7ff860195f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff8663a1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860101820 0x7ff86019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff8663a1700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86019d560 con 0x7ff8600fdec0 2023-07-14T19:02:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff8663a1700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86019d6a0 con 0x7ff860100e90 2023-07-14T19:02:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff8663a1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff86019d830 con 0x7ff860101820 2023-07-14T19:02:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8600fdec0 0x7ff8601959e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:48.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85ffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8600fdec0 0x7ff8601959e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34882/0 (socket says 172.21.15.78:34882) 2023-07-14T19:02:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85ffff700 1 -- 172.21.15.78:0/2506831534 learned_addr learned my addr 172.21.15.78:0/2506831534 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff86493e700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860101820 0x7ff86019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85f7fe700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff860100e90 0x7ff860195f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85ffff700 1 -- 172.21.15.78:0/2506831534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860101820 msgr2=0x7ff86019cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85ffff700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860101820 0x7ff86019cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85ffff700 1 -- 172.21.15.78:0/2506831534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff860100e90 msgr2=0x7ff860195f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85ffff700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff860100e90 0x7ff860195f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:48.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85ffff700 1 -- 172.21.15.78:0/2506831534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff858009d00 con 0x7ff8600fdec0 2023-07-14T19:02:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.325+0000 7ff85f7fe700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff860100e90 0x7ff860195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.326+0000 7ff85ffff700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8600fdec0 0x7ff8601959e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff85000d980 tx=0x7ff85000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.326+0000 7ff85d7fa700 1 -- 172.21.15.78:0/2506831534 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff850004020 con 0x7ff8600fdec0 2023-07-14T19:02:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.326+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff86019db10 con 0x7ff8600fdec0 2023-07-14T19:02:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.326+0000 7ff85d7fa700 1 -- 172.21.15.78:0/2506831534 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff850009730 con 0x7ff8600fdec0 2023-07-14T19:02:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.326+0000 7ff85d7fa700 1 -- 172.21.15.78:0/2506831534 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8500039c0 con 0x7ff8600fdec0 2023-07-14T19:02:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.326+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff86019e070 con 0x7ff8600fdec0 2023-07-14T19:02:48.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.327+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff860106bd0 con 0x7ff8600fdec0 2023-07-14T19:02:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.328+0000 7ff85d7fa700 1 -- 172.21.15.78:0/2506831534 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7ff8500098a0 con 0x7ff8600fdec0 2023-07-14T19:02:48.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.328+0000 7ff85d7fa700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff8480a1150 0x7ff8480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.328+0000 7ff85f7fe700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff8480a1150 0x7ff8480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.328+0000 7ff85d7fa700 1 -- 172.21.15.78:0/2506831534 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7ff8500ccb40 con 0x7ff8600fdec0 2023-07-14T19:02:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.329+0000 7ff85f7fe700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff8480a1150 0x7ff8480a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff854005950 tx=0x7ff854005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:48.336 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.332+0000 7ff85d7fa700 1 -- 172.21.15.78:0/2506831534 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff8500960a0 con 0x7ff8600fdec0 2023-07-14T19:02:48.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:48 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/866254233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:48.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:48 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1676524571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:02:48.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:48 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/866254233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:48.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:48 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1676524571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:02:48.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:48.564+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) v1 -- 0x7ff86002cb50 con 0x7ff8600fdec0 2023-07-14T19:02:48.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:48 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/866254233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:48.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:48 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1676524571' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2023-07-14T19:02:49.186 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.183+0000 7ff85d7fa700 1 -- 172.21.15.78:0/2506831534 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) v1 ==== 105+0+0 (secure 0 0 0) 0x7ff8500996c0 con 0x7ff8600fdec0 2023-07-14T19:02:49.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff8480a1150 msgr2=0x7ff8480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:49.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff8480a1150 0x7ff8480a3600 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ff854005950 tx=0x7ff854005f50 comp rx=0 tx=0).stop 2023-07-14T19:02:49.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8600fdec0 msgr2=0x7ff8601959e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:49.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8600fdec0 0x7ff8601959e0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ff85000d980 tx=0x7ff85000dc90 comp rx=0 tx=0).stop 2023-07-14T19:02:49.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 shutdown_connections 2023-07-14T19:02:49.190 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff8480a1150 0x7ff8480a3600 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff860101820 0x7ff86019cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8600fdec0 0x7ff8601959e0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 --2- 172.21.15.78:0/2506831534 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff860100e90 0x7ff860195f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 >> 172.21.15.78:0/2506831534 conn(0x7ff8600f98a0 msgr2=0x7ff8600fbb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:49.191 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 shutdown_connections 2023-07-14T19:02:49.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.186+0000 7ff8663a1700 1 -- 172.21.15.78:0/2506831534 wait complete. 2023-07-14T19:02:49.207 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2023-07-14T19:02:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 -- 172.21.15.78:0/1104816604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77200ff770 msgr2=0x7f7720107020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 --2- 172.21.15.78:0/1104816604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77200ff770 0x7f7720107020 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f771800b0a0 tx=0x7f771800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 -- 172.21.15.78:0/1104816604 shutdown_connections 2023-07-14T19:02:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 --2- 172.21.15.78:0/1104816604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77201000d0 0x7f77201004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 --2- 172.21.15.78:0/1104816604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77200ff770 0x7f7720107020 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 --2- 172.21.15.78:0/1104816604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77200fedc0 0x7f77200ff230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 -- 172.21.15.78:0/1104816604 >> 172.21.15.78:0/1104816604 conn(0x7f77200f98a0 msgr2=0x7f77200fbcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 -- 172.21.15.78:0/1104816604 shutdown_connections 2023-07-14T19:02:49.340 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.334+0000 7f7727867700 1 -- 172.21.15.78:0/1104816604 wait complete. 2023-07-14T19:02:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7727867700 1 Processor -- start 2023-07-14T19:02:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7727867700 1 -- start start 2023-07-14T19:02:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7727867700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77200fedc0 0x7f7720194890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7727867700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77200ff770 0x7f7720194dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7727867700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77201000d0 0x7f772019be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7727867700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772019c410 con 0x7f77201000d0 2023-07-14T19:02:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7727867700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772019c550 con 0x7f77200ff770 2023-07-14T19:02:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7727867700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f772019c6e0 con 0x7f77200fedc0 2023-07-14T19:02:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7725e04700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77201000d0 0x7f772019be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7725e04700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77201000d0 0x7f772019be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34900/0 (socket says 172.21.15.78:34900) 2023-07-14T19:02:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7725e04700 1 -- 172.21.15.78:0/996319189 learned_addr learned my addr 172.21.15.78:0/996319189 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.335+0000 7f7725603700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77200fedc0 0x7f7720194890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7724e02700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77200ff770 0x7f7720194dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7725e04700 1 -- 172.21.15.78:0/996319189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77200fedc0 msgr2=0x7f7720194890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7725e04700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77200fedc0 0x7f7720194890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7725e04700 1 -- 172.21.15.78:0/996319189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77200ff770 msgr2=0x7f7720194dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7725e04700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77200ff770 0x7f7720194dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7725e04700 1 -- 172.21.15.78:0/996319189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7718009d00 con 0x7f77201000d0 2023-07-14T19:02:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7725e04700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77201000d0 0x7f772019be40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f771800bad0 tx=0x7f771800bb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f77167fc700 1 -- 172.21.15.78:0/996319189 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f771800e040 con 0x7f77201000d0 2023-07-14T19:02:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f77167fc700 1 -- 172.21.15.78:0/996319189 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7718003c00 con 0x7f77201000d0 2023-07-14T19:02:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f772019c960 con 0x7f77201000d0 2023-07-14T19:02:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f77167fc700 1 -- 172.21.15.78:0/996319189 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77180052a0 con 0x7f77201000d0 2023-07-14T19:02:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.336+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f772019cde0 con 0x7f77201000d0 2023-07-14T19:02:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.337+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f772010a1c0 con 0x7f77201000d0 2023-07-14T19:02:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.338+0000 7f77167fc700 1 -- 172.21.15.78:0/996319189 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f771801d470 con 0x7f77201000d0 2023-07-14T19:02:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.339+0000 7f77167fc700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f770c0a1150 0x7f770c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.339+0000 7f7725603700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f770c0a1150 0x7f770c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.339+0000 7f77167fc700 1 -- 172.21.15.78:0/996319189 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f77180cde30 con 0x7f77201000d0 2023-07-14T19:02:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.339+0000 7f7725603700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f770c0a1150 0x7f770c0a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f77200ffdf0 tx=0x7f7710007400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.343+0000 7f77167fc700 1 -- 172.21.15.78:0/996319189 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f77180973c0 con 0x7f77201000d0 2023-07-14T19:02:49.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:49 smithi088 ceph-mon[128973]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-14T19:02:49.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:49 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1676524571' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:02:49.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:49 smithi088 ceph-mon[128973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:49.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:49 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2506831534' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:02:49.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:49 smithi078 ceph-mon[127425]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-14T19:02:49.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:49 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1676524571' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:02:49.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:49 smithi078 ceph-mon[127425]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:49.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:49 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2506831534' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:02:49.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:49.576+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f7720105ac0 con 0x7f77201000d0 2023-07-14T19:02:49.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:49 smithi102 ceph-mon[129280]: pgmap v264: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2023-07-14T19:02:49.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:49 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1676524571' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2023-07-14T19:02:49.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:49 smithi102 ceph-mon[129280]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:49.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:49 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2506831534' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2023-07-14T19:02:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.193+0000 7f77167fc700 1 -- 172.21.15.78:0/996319189 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) v1 ==== 121+0+0 (secure 0 0 0) 0x7f7718024350 con 0x7f77201000d0 2023-07-14T19:02:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f770c0a1150 msgr2=0x7f770c0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f770c0a1150 0x7f770c0a3600 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f77200ffdf0 tx=0x7f7710007400 comp rx=0 tx=0).stop 2023-07-14T19:02:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77201000d0 msgr2=0x7f772019be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77201000d0 0x7f772019be40 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f771800bad0 tx=0x7f771800bb00 comp rx=0 tx=0).stop 2023-07-14T19:02:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 shutdown_connections 2023-07-14T19:02:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f770c0a1150 0x7f770c0a3600 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77201000d0 0x7f772019be40 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77200fedc0 0x7f7720194890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 --2- 172.21.15.78:0/996319189 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77200ff770 0x7f7720194dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 >> 172.21.15.78:0/996319189 conn(0x7f77200f98a0 msgr2=0x7f77200fbab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 shutdown_connections 2023-07-14T19:02:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.196+0000 7f7727867700 1 -- 172.21.15.78:0/996319189 wait complete. 2023-07-14T19:02:50.217 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2023-07-14T19:02:50.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 -- 172.21.15.78:0/809407457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 msgr2=0x7f31b0100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 --2- 172.21.15.78:0/809407457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 0x7f31b0100030 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009990 tx=0x7f31a0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 -- 172.21.15.78:0/809407457 shutdown_connections 2023-07-14T19:02:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 --2- 172.21.15.78:0/809407457 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31b01017a0 0x7f31b0108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 --2- 172.21.15.78:0/809407457 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 0x7f31b0100030 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 --2- 172.21.15.78:0/809407457 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31b0100e10 0x7f31b0101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 -- 172.21.15.78:0/809407457 >> 172.21.15.78:0/809407457 conn(0x7f31b00fb260 msgr2=0x7f31b00fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 -- 172.21.15.78:0/809407457 shutdown_connections 2023-07-14T19:02:50.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.341+0000 7f31b5219700 1 -- 172.21.15.78:0/809407457 wait complete. 2023-07-14T19:02:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.342+0000 7f31b5219700 1 Processor -- start 2023-07-14T19:02:50.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.342+0000 7f31b5219700 1 -- start start 2023-07-14T19:02:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.342+0000 7f31b5219700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 0x7f31b0197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.342+0000 7f31b5219700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31b0100e10 0x7f31b0197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.342+0000 7f31b5219700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31b01017a0 0x7f31b019f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.342+0000 7f31b5219700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b019f5d0 con 0x7f31b00ffc20 2023-07-14T19:02:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.342+0000 7f31b5219700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b019f710 con 0x7f31b01017a0 2023-07-14T19:02:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.342+0000 7f31b5219700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b019f8a0 con 0x7f31b0100e10 2023-07-14T19:02:50.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 0x7f31b0197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 0x7f31b0197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34910/0 (socket says 172.21.15.78:34910) 2023-07-14T19:02:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 -- 172.21.15.78:0/95155544 learned_addr learned my addr 172.21.15.78:0/95155544 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31ae59c700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31b0100e10 0x7f31b0197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31af59e700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31b01017a0 0x7f31b019f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 -- 172.21.15.78:0/95155544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31b0100e10 msgr2=0x7f31b0197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31b0100e10 0x7f31b0197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 -- 172.21.15.78:0/95155544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31b01017a0 msgr2=0x7f31b019f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31b01017a0 0x7f31b019f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 -- 172.21.15.78:0/95155544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31a00095f0 con 0x7f31b00ffc20 2023-07-14T19:02:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31aed9d700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 0x7f31b0197a50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f31a000b5c0 tx=0x7f31a0005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f3197fff700 1 -- 172.21.15.78:0/95155544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a001d070 con 0x7f31b00ffc20 2023-07-14T19:02:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f3197fff700 1 -- 172.21.15.78:0/95155544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f31a0004000 con 0x7f31b00ffc20 2023-07-14T19:02:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b019fb20 con 0x7f31b00ffc20 2023-07-14T19:02:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f3197fff700 1 -- 172.21.15.78:0/95155544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a0022d00 con 0x7f31b00ffc20 2023-07-14T19:02:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.343+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b019ff40 con 0x7f31b00ffc20 2023-07-14T19:02:50.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.344+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b0192160 con 0x7f31b00ffc20 2023-07-14T19:02:50.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.345+0000 7f3197fff700 1 -- 172.21.15.78:0/95155544 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f31a0003b70 con 0x7f31b00ffc20 2023-07-14T19:02:50.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.346+0000 7f3197fff700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f31980a1150 0x7f31980a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:50.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.346+0000 7f31ae59c700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f31980a1150 0x7f31980a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:50.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.346+0000 7f3197fff700 1 -- 172.21.15.78:0/95155544 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f31a00ce5b0 con 0x7f31b00ffc20 2023-07-14T19:02:50.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.346+0000 7f31ae59c700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f31980a1150 0x7f31980a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f31a4007c60 tx=0x7f31a40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:50.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.349+0000 7f3197fff700 1 -- 172.21.15.78:0/95155544 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f31a0097ba0 con 0x7f31b00ffc20 2023-07-14T19:02:50.462 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:02:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b[154122]: 2023-07-14T19:02:50.403+0000 7fa961a23700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2023-07-14T19:02:50.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:50 smithi088 ceph-mon[128973]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:02:50.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:50 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2506831534' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:02:50.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:50 smithi088 ceph-mon[128973]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:02:50.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:50 smithi088 ceph-mon[128973]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:50.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:50 smithi088 ceph-mon[128973]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:02:50.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:50 smithi088 ceph-mon[128973]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:02:50.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:50 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/996319189' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:02:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-mon[127425]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:02:50.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2506831534' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:02:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-mon[127425]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:02:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-mon[127425]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-mon[127425]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:02:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-mon[127425]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:02:50.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/996319189' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:02:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.588+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f31b002c9d0 con 0x7f31b00ffc20 2023-07-14T19:02:50.612 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.608+0000 7f3197fff700 1 -- 172.21.15.78:0/95155544 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) v1 ==== 115+0+0 (secure 0 0 0) 0x7f31a0022e60 con 0x7f31b00ffc20 2023-07-14T19:02:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f31980a1150 msgr2=0x7f31980a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f31980a1150 0x7f31980a3600 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f31a4007c60 tx=0x7f31a40073d0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 msgr2=0x7f31b0197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 0x7f31b0197a50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f31a000b5c0 tx=0x7f31a0005740 comp rx=0 tx=0).stop 2023-07-14T19:02:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 shutdown_connections 2023-07-14T19:02:50.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f31980a1150 0x7f31980a3600 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f31b01017a0 0x7f31b019f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f31b00ffc20 0x7f31b0197a50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 --2- 172.21.15.78:0/95155544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31b0100e10 0x7f31b0197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 >> 172.21.15.78:0/95155544 conn(0x7f31b00fb260 msgr2=0x7f31b0106750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:50.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 shutdown_connections 2023-07-14T19:02:50.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.611+0000 7f31b5219700 1 -- 172.21.15.78:0/95155544 wait complete. 2023-07-14T19:02:50.628 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2023-07-14T19:02:50.628 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 '' 2023-07-14T19:02:50.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:50 smithi102 ceph-mon[129280]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2023-07-14T19:02:50.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:50 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2506831534' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2023-07-14T19:02:50.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:50 smithi102 ceph-mon[129280]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2023-07-14T19:02:50.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:50 smithi102 ceph-mon[129280]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2023-07-14T19:02:50.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:50 smithi102 ceph-mon[129280]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2023-07-14T19:02:50.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:50 smithi102 ceph-mon[129280]: daemon mds.i is now active in filesystem cephfs as rank 1 2023-07-14T19:02:50.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:50 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/996319189' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2023-07-14T19:02:50.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 -- 172.21.15.78:0/2188196287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 msgr2=0x7f7460100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 --2- 172.21.15.78:0/2188196287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 0x7f7460100500 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7454009990 tx=0x7f7454009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.754 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 -- 172.21.15.78:0/2188196287 shutdown_connections 2023-07-14T19:02:50.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 --2- 172.21.15.78:0/2188196287 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74600ff6f0 0x7f7460106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 --2- 172.21.15.78:0/2188196287 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 0x7f7460100500 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 --2- 172.21.15.78:0/2188196287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74600fed40 0x7f74600ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 -- 172.21.15.78:0/2188196287 >> 172.21.15.78:0/2188196287 conn(0x7f74600f98e0 msgr2=0x7f74600fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:50.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 -- 172.21.15.78:0/2188196287 shutdown_connections 2023-07-14T19:02:50.755 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.750+0000 7f7464ec3700 1 -- 172.21.15.78:0/2188196287 wait complete. 2023-07-14T19:02:50.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 Processor -- start 2023-07-14T19:02:50.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 -- start start 2023-07-14T19:02:50.757 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74600fed40 0x7f74600fe800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74600ff6f0 0x7f746019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 0x7f746019cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746019d340 con 0x7f74601000f0 2023-07-14T19:02:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746019d4b0 con 0x7f74600fed40 2023-07-14T19:02:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f746019d670 con 0x7f74600ff6f0 2023-07-14T19:02:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 0x7f746019cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 0x7f746019cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34928/0 (socket says 172.21.15.78:34928) 2023-07-14T19:02:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 -- 172.21.15.78:0/4194894037 learned_addr learned my addr 172.21.15.78:0/4194894037 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745dd9b700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74600ff6f0 0x7f746019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745e59c700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74600fed40 0x7f74600fe800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 -- 172.21.15.78:0/4194894037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74600ff6f0 msgr2=0x7f746019a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74600ff6f0 0x7f746019a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 -- 172.21.15.78:0/4194894037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74600fed40 msgr2=0x7f74600fe800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74600fed40 0x7f74600fe800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 -- 172.21.15.78:0/4194894037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74540095f0 con 0x7f74601000f0 2023-07-14T19:02:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f745ed9d700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 0x7f746019cba0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f745800b970 tx=0x7f745800bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f744f7fe700 1 -- 172.21.15.78:0/4194894037 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f745800c9b0 con 0x7f74601000f0 2023-07-14T19:02:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f744f7fe700 1 -- 172.21.15.78:0/4194894037 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7458014440 con 0x7f74601000f0 2023-07-14T19:02:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f746019d950 con 0x7f74601000f0 2023-07-14T19:02:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.751+0000 7f744f7fe700 1 -- 172.21.15.78:0/4194894037 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580075d0 con 0x7f74601000f0 2023-07-14T19:02:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.752+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f746019de90 con 0x7f74601000f0 2023-07-14T19:02:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.752+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f746010a220 con 0x7f74601000f0 2023-07-14T19:02:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.753+0000 7f744f7fe700 1 -- 172.21.15.78:0/4194894037 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f74580145b0 con 0x7f74601000f0 2023-07-14T19:02:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.754+0000 7f744f7fe700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74480a1150 0x7f74480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.754+0000 7f745e59c700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74480a1150 0x7f74480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.754+0000 7f744f7fe700 1 -- 172.21.15.78:0/4194894037 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7458006e40 con 0x7f74601000f0 2023-07-14T19:02:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.755+0000 7f745e59c700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74480a1150 0x7f74480a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f745400b5c0 tx=0x7f7454005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.758+0000 7f744f7fe700 1 -- 172.21.15.78:0/4194894037 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f74580960a0 con 0x7f74601000f0 2023-07-14T19:02:50.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:50.976+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f7460060a80 con 0x7f74480a1150 2023-07-14T19:02:51.022 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.018+0000 7f744f7fe700 1 -- 172.21.15.78:0/4194894037 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7460060a80 con 0x7f74480a1150 2023-07-14T19:02:51.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.020+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74480a1150 msgr2=0x7f74480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.020+0000 7f7464ec3700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74480a1150 0x7f74480a3600 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f745400b5c0 tx=0x7f7454005fb0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.020+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 msgr2=0x7f746019cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.020+0000 7f7464ec3700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 0x7f746019cba0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f745800b970 tx=0x7f745800bd30 comp rx=0 tx=0).stop 2023-07-14T19:02:51.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.021+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 shutdown_connections 2023-07-14T19:02:51.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.021+0000 7f7464ec3700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74480a1150 0x7f74480a3600 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.021+0000 7f7464ec3700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74600fed40 0x7f74600fe800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.021+0000 7f7464ec3700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74601000f0 0x7f746019cba0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.021+0000 7f7464ec3700 1 --2- 172.21.15.78:0/4194894037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74600ff6f0 0x7f746019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.021+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 >> 172.21.15.78:0/4194894037 conn(0x7f74600f98e0 msgr2=0x7f74600fbb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:51.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.021+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 shutdown_connections 2023-07-14T19:02:51.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.021+0000 7f7464ec3700 1 -- 172.21.15.78:0/4194894037 wait complete. 2023-07-14T19:02:51.037 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_0 2023-07-14T19:02:51.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- 172.21.15.78:0/3892162948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80fedc0 msgr2=0x7f30f80ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 --2- 172.21.15.78:0/3892162948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80fedc0 0x7f30f80ff210 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f30e4009990 tx=0x7f30e4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- 172.21.15.78:0/3892162948 shutdown_connections 2023-07-14T19:02:51.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 --2- 172.21.15.78:0/3892162948 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30f80ff750 0x7f30f8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 --2- 172.21.15.78:0/3892162948 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30f8100090 0x7f30f81004a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 --2- 172.21.15.78:0/3892162948 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80fedc0 0x7f30f80ff210 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- 172.21.15.78:0/3892162948 >> 172.21.15.78:0/3892162948 conn(0x7f30f80f98a0 msgr2=0x7f30f80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:51.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- 172.21.15.78:0/3892162948 shutdown_connections 2023-07-14T19:02:51.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- 172.21.15.78:0/3892162948 wait complete. 2023-07-14T19:02:51.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 Processor -- start 2023-07-14T19:02:51.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- start start 2023-07-14T19:02:51.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30f80fedc0 0x7f30f8102b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80ff750 0x7f30f81011b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30f8100090 0x7f30f81016f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8101e90 con 0x7f30f80ff750 2023-07-14T19:02:51.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f8102000 con 0x7f30f8100090 2023-07-14T19:02:51.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.157+0000 7f30feec4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30f819f8c0 con 0x7f30f80fedc0 2023-07-14T19:02:51.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80ff750 0x7f30f81011b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80ff750 0x7f30f81011b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34960/0 (socket says 172.21.15.78:34960) 2023-07-14T19:02:51.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 -- 172.21.15.78:0/626735575 learned_addr learned my addr 172.21.15.78:0/626735575 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:51.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30fcc60700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30f80fedc0 0x7f30f8102b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30fd461700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30f8100090 0x7f30f81016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 -- 172.21.15.78:0/626735575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30f80fedc0 msgr2=0x7f30f8102b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30f80fedc0 0x7f30f8102b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 -- 172.21.15.78:0/626735575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30f8100090 msgr2=0x7f30f81016f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30f8100090 0x7f30f81016f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 -- 172.21.15.78:0/626735575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30e40095f0 con 0x7f30f80ff750 2023-07-14T19:02:51.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30fcc60700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30f80fedc0 0x7f30f8102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:51.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f7fff700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80ff750 0x7f30f81011b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f30e4006010 tx=0x7f30e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:51.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f5ffb700 1 -- 172.21.15.78:0/626735575 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e401d070 con 0x7f30f80ff750 2023-07-14T19:02:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30f819fb40 con 0x7f30f80ff750 2023-07-14T19:02:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f5ffb700 1 -- 172.21.15.78:0/626735575 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f30e40041b0 con 0x7f30f80ff750 2023-07-14T19:02:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30f5ffb700 1 -- 172.21.15.78:0/626735575 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30e4022da0 con 0x7f30f80ff750 2023-07-14T19:02:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.158+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30f819ffc0 con 0x7f30f80ff750 2023-07-14T19:02:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.159+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30f810a170 con 0x7f30f80ff750 2023-07-14T19:02:51.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.160+0000 7f30f5ffb700 1 -- 172.21.15.78:0/626735575 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f30e4022570 con 0x7f30f80ff750 2023-07-14T19:02:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.161+0000 7f30f5ffb700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f30e80a1150 0x7f30e80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.161+0000 7f30f5ffb700 1 -- 172.21.15.78:0/626735575 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f30e40ce4e0 con 0x7f30f80ff750 2023-07-14T19:02:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.161+0000 7f30fcc60700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f30e80a1150 0x7f30e80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.161+0000 7f30fcc60700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f30e80a1150 0x7f30e80a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f30ec005950 tx=0x7f30ec0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:51.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.164+0000 7f30f5ffb700 1 -- 172.21.15.78:0/626735575 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f30e4097a70 con 0x7f30f80ff750 2023-07-14T19:02:51.297 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:02:50.981+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:02:51.298 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:02:50.981+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:02:51.298 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:02:50.981+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:02:51.298 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:02:50.981+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:02:51.298 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:02:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:02:50.981+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:02:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-07-14T19:02:51.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] up:active 2023-07-14T19:02:51.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/996319189' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:02:51.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:02:51.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:02:51.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: Cluster is now healthy 2023-07-14T19:02:51.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:02:51.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:02:51.299 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:02:51.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/95155544' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:02:51.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] up:active 2023-07-14T19:02:51.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/95155544' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:02:51.300 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:51 smithi078 ceph-mon[127425]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:02:51.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.388+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f30f80fe3c0 con 0x7f30e80a1150 2023-07-14T19:02:51.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.392+0000 7f30f5ffb700 1 -- 172.21.15.78:0/626735575 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f30f80fe3c0 con 0x7f30e80a1150 2023-07-14T19:02:51.396 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/_nogroup/sv_0/9431b327-2d1c-4178-bfca-51125bd3288c 2023-07-14T19:02:51.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f30e80a1150 msgr2=0x7f30e80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f30e80a1150 0x7f30e80a3600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f30ec005950 tx=0x7f30ec0058e0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80ff750 msgr2=0x7f30f81011b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80ff750 0x7f30f81011b0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f30e4006010 tx=0x7f30e4004730 comp rx=0 tx=0).stop 2023-07-14T19:02:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 shutdown_connections 2023-07-14T19:02:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f30e80a1150 0x7f30e80a3600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f30f8100090 0x7f30f81016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30f80fedc0 0x7f30f8102b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 --2- 172.21.15.78:0/626735575 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f30f80ff750 0x7f30f81011b0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 >> 172.21.15.78:0/626735575 conn(0x7f30f80f98a0 msgr2=0x7f30f80fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:51.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 shutdown_connections 2023-07-14T19:02:51.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.395+0000 7f30feec4700 1 -- 172.21.15.78:0/626735575 wait complete. 2023-07-14T19:02:51.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 '' 2023-07-14T19:02:51.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-07-14T19:02:51.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] up:active 2023-07-14T19:02:51.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/996319189' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:02:51.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:02:51.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:02:51.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: Cluster is now healthy 2023-07-14T19:02:51.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:02:51.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:02:51.464 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:02:51.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/95155544' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:02:51.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] up:active 2023-07-14T19:02:51.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/95155544' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:02:51.465 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:51 smithi088 ceph-mon[128973]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:02:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.531+0000 7f5c5397c700 1 -- 172.21.15.78:0/3017165674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c1017c0 msgr2=0x7f5c4c108080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.531+0000 7f5c5397c700 1 --2- 172.21.15.78:0/3017165674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c1017c0 0x7f5c4c108080 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400b0a0 tx=0x7f5c4400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 -- 172.21.15.78:0/3017165674 shutdown_connections 2023-07-14T19:02:51.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 --2- 172.21.15.78:0/3017165674 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c1017c0 0x7f5c4c108080 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 --2- 172.21.15.78:0/3017165674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c4c0ffc40 0x7f5c4c100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 --2- 172.21.15.78:0/3017165674 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c100e30 0x7f5c4c101280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 -- 172.21.15.78:0/3017165674 >> 172.21.15.78:0/3017165674 conn(0x7f5c4c0fb240 msgr2=0x7f5c4c0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 -- 172.21.15.78:0/3017165674 shutdown_connections 2023-07-14T19:02:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 -- 172.21.15.78:0/3017165674 wait complete. 2023-07-14T19:02:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 Processor -- start 2023-07-14T19:02:51.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 -- start start 2023-07-14T19:02:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c4c0ffc40 0x7f5c4c195930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c100e30 0x7f5c4c195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1017c0 0x7f5c4c19cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c19d4b0 con 0x7f5c4c100e30 2023-07-14T19:02:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c19d5f0 con 0x7f5c4c1017c0 2023-07-14T19:02:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c5397c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c4c19d780 con 0x7f5c4c0ffc40 2023-07-14T19:02:51.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c50f17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c100e30 0x7f5c4c195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.532+0000 7f5c50f17700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c100e30 0x7f5c4c195e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34974/0 (socket says 172.21.15.78:34974) 2023-07-14T19:02:51.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c50f17700 1 -- 172.21.15.78:0/4143198909 learned_addr learned my addr 172.21.15.78:0/4143198909 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:51.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c51f19700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1017c0 0x7f5c4c19cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c51718700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c4c0ffc40 0x7f5c4c195930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c50f17700 1 -- 172.21.15.78:0/4143198909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c4c0ffc40 msgr2=0x7f5c4c195930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c50f17700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c4c0ffc40 0x7f5c4c195930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c50f17700 1 -- 172.21.15.78:0/4143198909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1017c0 msgr2=0x7f5c4c19cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c50f17700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1017c0 0x7f5c4c19cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c50f17700 1 -- 172.21.15.78:0/4143198909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c44009d00 con 0x7f5c4c100e30 2023-07-14T19:02:51.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c50f17700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c100e30 0x7f5c4c195e70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00d980 tx=0x7f5c3c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:51.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c427fc700 1 -- 172.21.15.78:0/4143198909 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c004220 con 0x7f5c4c100e30 2023-07-14T19:02:51.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c4c19da60 con 0x7f5c4c100e30 2023-07-14T19:02:51.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c427fc700 1 -- 172.21.15.78:0/4143198909 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5c3c007640 con 0x7f5c4c100e30 2023-07-14T19:02:51.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c427fc700 1 -- 172.21.15.78:0/4143198909 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3c0055e0 con 0x7f5c4c100e30 2023-07-14T19:02:51.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.533+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c4c19df70 con 0x7f5c4c100e30 2023-07-14T19:02:51.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.534+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4c1070d0 con 0x7f5c4c100e30 2023-07-14T19:02:51.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.535+0000 7f5c427fc700 1 -- 172.21.15.78:0/4143198909 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f5c3c004750 con 0x7f5c4c100e30 2023-07-14T19:02:51.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.535+0000 7f5c427fc700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5c380a1150 0x7f5c380a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.536+0000 7f5c427fc700 1 -- 172.21.15.78:0/4143198909 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f5c3c0cc910 con 0x7f5c4c100e30 2023-07-14T19:02:51.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.536+0000 7f5c51718700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5c380a1150 0x7f5c380a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.536+0000 7f5c51718700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5c380a1150 0x7f5c380a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5c48005fd0 tx=0x7f5c48005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:51.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.540+0000 7f5c427fc700 1 -- 172.21.15.78:0/4143198909 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5c3c095e70 con 0x7f5c4c100e30 2023-07-14T19:02:51.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: pgmap v265: 129 pgs: 129 active+clean; 581 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 0 op/s 2023-07-14T19:02:51.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] up:active 2023-07-14T19:02:51.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/996319189' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2023-07-14T19:02:51.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2023-07-14T19:02:51.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2023-07-14T19:02:51.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: Cluster is now healthy 2023-07-14T19:02:51.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2023-07-14T19:02:51.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2023-07-14T19:02:51.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: daemon mds.h is now active in filesystem cephfs as rank 2 2023-07-14T19:02:51.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/95155544' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2023-07-14T19:02:51.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] up:active 2023-07-14T19:02:51.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/95155544' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2023-07-14T19:02:51.635 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:51 smithi102 ceph-mon[129280]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:02:51.760 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.756+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c4c060a80 con 0x7f5c380a1150 2023-07-14T19:02:51.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.781+0000 7f5c427fc700 1 -- 172.21.15.78:0/4143198909 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f5c4c060a80 con 0x7f5c380a1150 2023-07-14T19:02:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.782+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5c380a1150 msgr2=0x7f5c380a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.782+0000 7f5c5397c700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5c380a1150 0x7f5c380a3600 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5c48005fd0 tx=0x7f5c48005e50 comp rx=0 tx=0).stop 2023-07-14T19:02:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.782+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c100e30 msgr2=0x7f5c4c195e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c100e30 0x7f5c4c195e70 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5c3c00d980 tx=0x7f5c3c00dc90 comp rx=0 tx=0).stop 2023-07-14T19:02:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 shutdown_connections 2023-07-14T19:02:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5c380a1150 0x7f5c380a3600 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5c4c1017c0 0x7f5c4c19cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5c4c0ffc40 0x7f5c4c195930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 --2- 172.21.15.78:0/4143198909 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5c4c100e30 0x7f5c4c195e70 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 >> 172.21.15.78:0/4143198909 conn(0x7f5c4c0fb240 msgr2=0x7f5c4c104210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 shutdown_connections 2023-07-14T19:02:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.783+0000 7f5c5397c700 1 -- 172.21.15.78:0/4143198909 wait complete. 2023-07-14T19:02:51.798 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs sv_1 2023-07-14T19:02:51.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 -- 172.21.15.78:0/3004063903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c100690 msgr2=0x7f2c9c102a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 --2- 172.21.15.78:0/3004063903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c100690 0x7f2c9c102a70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c009990 tx=0x7f2c8c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 -- 172.21.15.78:0/3004063903 shutdown_connections 2023-07-14T19:02:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 --2- 172.21.15.78:0/3004063903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c9c1058d0 0x7f2c9c107d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 --2- 172.21.15.78:0/3004063903 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c100690 0x7f2c9c102a70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 --2- 172.21.15.78:0/3004063903 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c102fb0 0x7f2c9c105390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 -- 172.21.15.78:0/3004063903 >> 172.21.15.78:0/3004063903 conn(0x7f2c9c0fa130 msgr2=0x7f2c9c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:51.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 -- 172.21.15.78:0/3004063903 shutdown_connections 2023-07-14T19:02:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.918+0000 7f2ca2333700 1 -- 172.21.15.78:0/3004063903 wait complete. 2023-07-14T19:02:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca2333700 1 Processor -- start 2023-07-14T19:02:51.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca2333700 1 -- start start 2023-07-14T19:02:51.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca2333700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c100690 0x7f2c9c195850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca2333700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c9c102fb0 0x7f2c9c195d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca2333700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c1058d0 0x7f2c9c19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca2333700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c19d3d0 con 0x7f2c9c1058d0 2023-07-14T19:02:51.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca2333700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c19d510 con 0x7f2c9c100690 2023-07-14T19:02:51.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca2333700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c19d6a0 con 0x7f2c9c102fb0 2023-07-14T19:02:51.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca08d0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c1058d0 0x7f2c9c19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca08d0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c1058d0 0x7f2c9c19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:34992/0 (socket says 172.21.15.78:34992) 2023-07-14T19:02:51.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2c9bfff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c100690 0x7f2c9c195850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca08d0700 1 -- 172.21.15.78:0/1611663022 learned_addr learned my addr 172.21.15.78:0/1611663022 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2c9b7fe700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c9c102fb0 0x7f2c9c195d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca08d0700 1 -- 172.21.15.78:0/1611663022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c9c102fb0 msgr2=0x7f2c9c195d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca08d0700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c9c102fb0 0x7f2c9c195d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca08d0700 1 -- 172.21.15.78:0/1611663022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c100690 msgr2=0x7f2c9c195850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:51.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca08d0700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c100690 0x7f2c9c195850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.919+0000 7f2ca08d0700 1 -- 172.21.15.78:0/1611663022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c8c0095f0 con 0x7f2c9c1058d0 2023-07-14T19:02:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.920+0000 7f2ca08d0700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c1058d0 0x7f2c9c19ce00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2c94007ee0 tx=0x7f2c9400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.920+0000 7f2c997fa700 1 -- 172.21.15.78:0/1611663022 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94017070 con 0x7f2c9c1058d0 2023-07-14T19:02:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.920+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c19d980 con 0x7f2c9c1058d0 2023-07-14T19:02:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.920+0000 7f2c997fa700 1 -- 172.21.15.78:0/1611663022 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2c9400f040 con 0x7f2c9c1058d0 2023-07-14T19:02:51.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.920+0000 7f2c997fa700 1 -- 172.21.15.78:0/1611663022 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c94005570 con 0x7f2c9c1058d0 2023-07-14T19:02:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.920+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c19de40 con 0x7f2c9c1058d0 2023-07-14T19:02:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.921+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c0fbca0 con 0x7f2c9c1058d0 2023-07-14T19:02:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.922+0000 7f2c997fa700 1 -- 172.21.15.78:0/1611663022 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 16) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f2c9400db20 con 0x7f2c9c1058d0 2023-07-14T19:02:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.922+0000 7f2c997fa700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c840a1150 0x7f2c840a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.922+0000 7f2c9bfff700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c840a1150 0x7f2c840a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.922+0000 7f2c997fa700 1 -- 172.21.15.78:0/1611663022 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2c940cc2b0 con 0x7f2c9c1058d0 2023-07-14T19:02:51.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.923+0000 7f2c9bfff700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c840a1150 0x7f2c840a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00b5c0 tx=0x7f2c8c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:51.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:51.926+0000 7f2c997fa700 1 -- 172.21.15.78:0/1611663022 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2c94095810 con 0x7f2c9c1058d0 2023-07-14T19:02:52.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.149+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2c9c060a80 con 0x7f2c840a1150 2023-07-14T19:02:52.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.153+0000 7f2c997fa700 1 -- 172.21.15.78:0/1611663022 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+60 (secure 0 0 0) 0x7f2c9c060a80 con 0x7f2c840a1150 2023-07-14T19:02:52.157 INFO:teuthology.orchestra.run.smithi078.stdout:/volumes/_nogroup/sv_1/5c09dadf-8e50-481d-8d05-7e714e4e25d3 2023-07-14T19:02:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c840a1150 msgr2=0x7f2c840a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c840a1150 0x7f2c840a3600 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2c8c00b5c0 tx=0x7f2c8c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c1058d0 msgr2=0x7f2c9c19ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c1058d0 0x7f2c9c19ce00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2c94007ee0 tx=0x7f2c9400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 shutdown_connections 2023-07-14T19:02:52.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c840a1150 0x7f2c840a3600 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c9c100690 0x7f2c9c195850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c9c1058d0 0x7f2c9c19ce00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 --2- 172.21.15.78:0/1611663022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c9c102fb0 0x7f2c9c195d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 >> 172.21.15.78:0/1611663022 conn(0x7f2c9c0fa130 msgr2=0x7f2c9c1012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:52.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 shutdown_connections 2023-07-14T19:02:52.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.155+0000 7f2ca2333700 1 -- 172.21.15.78:0/1611663022 wait complete. 2023-07-14T19:02:52.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:02:52.298 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 -- 172.21.15.78:0/3484822922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 msgr2=0x7f7adc106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3484822922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 0x7f7adc106f60 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7ad400b0a0 tx=0x7f7ad400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 -- 172.21.15.78:0/3484822922 shutdown_connections 2023-07-14T19:02:52.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3484822922 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 0x7f7adc106f60 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3484822922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7adc1000b0 0x7f7adc1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3484822922 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7adc0fed00 0x7f7adc0ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 -- 172.21.15.78:0/3484822922 >> 172.21.15.78:0/3484822922 conn(0x7f7adc0f98a0 msgr2=0x7f7adc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:52.299 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 -- 172.21.15.78:0/3484822922 shutdown_connections 2023-07-14T19:02:52.300 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.294+0000 7f7ae3468700 1 -- 172.21.15.78:0/3484822922 wait complete. 2023-07-14T19:02:52.301 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae3468700 1 Processor -- start 2023-07-14T19:02:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae3468700 1 -- start start 2023-07-14T19:02:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae3468700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7adc0fed00 0x7f7adc195900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae3468700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 0x7f7adc195e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae3468700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7adc1000b0 0x7f7adc19ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae3468700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc19d480 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.302 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae3468700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc19d5c0 con 0x7f7adc1000b0 2023-07-14T19:02:52.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae3468700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7adc19d750 con 0x7f7adc0fed00 2023-07-14T19:02:52.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae0a03700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 0x7f7adc195e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:52.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae0a03700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 0x7f7adc195e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35004/0 (socket says 172.21.15.78:35004) 2023-07-14T19:02:52.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.295+0000 7f7ae0a03700 1 -- 172.21.15.78:0/3705707604 learned_addr learned my addr 172.21.15.78:0/3705707604 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:52.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae1a05700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7adc1000b0 0x7f7adc19ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:52.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae1204700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7adc0fed00 0x7f7adc195900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:52.303 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae0a03700 1 -- 172.21.15.78:0/3705707604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7adc0fed00 msgr2=0x7f7adc195900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae0a03700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7adc0fed00 0x7f7adc195900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae0a03700 1 -- 172.21.15.78:0/3705707604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7adc1000b0 msgr2=0x7f7adc19ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae0a03700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7adc1000b0 0x7f7adc19ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae0a03700 1 -- 172.21.15.78:0/3705707604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ad4009d00 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae1a05700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7adc1000b0 0x7f7adc19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:52.304 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae0a03700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 0x7f7adc195e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800c920 tx=0x7f7ad800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:52.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ad27fc700 1 -- 172.21.15.78:0/3705707604 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8007b00 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7adc19da30 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ad27fc700 1 -- 172.21.15.78:0/3705707604 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7ad8004cc0 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ad27fc700 1 -- 172.21.15.78:0/3705707604 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ad8005490 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.296+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7adc19df40 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.305 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.297+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7adc106070 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.299+0000 7f7ad27fc700 1 -- 172.21.15.78:0/3705707604 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7ad8004750 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.299+0000 7f7ad27fc700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ac80a1220 0x7f7ac80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:52.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.299+0000 7f7ae1204700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ac80a1220 0x7f7ac80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:52.306 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.299+0000 7f7ad27fc700 1 -- 172.21.15.78:0/3705707604 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f7ad80ccc90 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.300+0000 7f7ae1204700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ac80a1220 0x7f7ac80a36d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005fd0 tx=0x7f7acc00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:52.307 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.302+0000 7f7ad27fc700 1 -- 172.21.15.78:0/3705707604 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7ad8096090 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:52 smithi088 ceph-mon[128973]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:52.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:52 smithi088 ceph-mon[128973]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:52.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.534+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7adc02c8e0 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.534+0000 7f7ad27fc700 1 -- 172.21.15.78:0/3705707604 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10808 (secure 0 0 0) 0x7f7ad801d020 con 0x7f7adc0ff6b0 2023-07-14T19:02:52.538 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:52.539 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:02:50.605732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24485":{"gid":24485,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:02:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.536+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ac80a1220 msgr2=0x7f7ac80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.536+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ac80a1220 0x7f7ac80a36d0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7acc005fd0 tx=0x7f7acc00a560 comp rx=0 tx=0).stop 2023-07-14T19:02:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.536+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 msgr2=0x7f7adc195e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.536+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 0x7f7adc195e40 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7ad800c920 tx=0x7f7ad800cce0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.537+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 shutdown_connections 2023-07-14T19:02:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.537+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ac80a1220 0x7f7ac80a36d0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.537+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7adc1000b0 0x7f7adc19ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.537+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7adc0fed00 0x7f7adc195900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.537+0000 7f7ae3468700 1 --2- 172.21.15.78:0/3705707604 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7adc0ff6b0 0x7f7adc195e40 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.537+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 >> 172.21.15.78:0/3705707604 conn(0x7f7adc0f98a0 msgr2=0x7f7adc1031b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:52.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.537+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 shutdown_connections 2023-07-14T19:02:52.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.537+0000 7f7ae3468700 1 -- 172.21.15.78:0/3705707604 wait complete. 2023-07-14T19:02:52.542 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2023-07-14T19:02:52.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:52 smithi078 ceph-mon[127425]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:52.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:52 smithi078 ceph-mon[127425]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:52.553 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:02:52.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:52 smithi102 ceph-mon[129280]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:52.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:52 smithi102 ceph-mon[129280]: from='client.14889 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:52.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 -- 172.21.15.78:0/3636634800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240ffd20 msgr2=0x7f8b24100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/3636634800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240ffd20 0x7f8b24100170 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8b20009990 tx=0x7f8b20009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 -- 172.21.15.78:0/3636634800 shutdown_connections 2023-07-14T19:02:52.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/3636634800 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b241006b0 0x7f8b24106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/3636634800 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b240feb30 0x7f8b240fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/3636634800 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b240ffd20 0x7f8b24100170 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 -- 172.21.15.78:0/3636634800 >> 172.21.15.78:0/3636634800 conn(0x7f8b240fa150 msgr2=0x7f8b240fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:52.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 -- 172.21.15.78:0/3636634800 shutdown_connections 2023-07-14T19:02:52.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.670+0000 7f8b2bd05700 1 -- 172.21.15.78:0/3636634800 wait complete. 2023-07-14T19:02:52.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2bd05700 1 Processor -- start 2023-07-14T19:02:52.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2bd05700 1 -- start start 2023-07-14T19:02:52.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2bd05700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b240feb30 0x7f8b24069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:52.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2bd05700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b240ffd20 0x7f8b2419a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:52.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2bd05700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b241006b0 0x7f8b2419cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:52.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2bd05700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2419d340 con 0x7f8b241006b0 2023-07-14T19:02:52.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2bd05700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2419d4b0 con 0x7f8b240ffd20 2023-07-14T19:02:52.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2bd05700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2419d670 con 0x7f8b240feb30 2023-07-14T19:02:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2a2a2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b241006b0 0x7f8b2419cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2a2a2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b241006b0 0x7f8b2419cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35020/0 (socket says 172.21.15.78:35020) 2023-07-14T19:02:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b2a2a2700 1 -- 172.21.15.78:0/26879894 learned_addr learned my addr 172.21.15.78:0/26879894 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b29aa1700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b240feb30 0x7f8b24069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.671+0000 7f8b292a0700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b240ffd20 0x7f8b2419a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b2a2a2700 1 -- 172.21.15.78:0/26879894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b240feb30 msgr2=0x7f8b24069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b2a2a2700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b240feb30 0x7f8b24069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b2a2a2700 1 -- 172.21.15.78:0/26879894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b240ffd20 msgr2=0x7f8b2419a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b2a2a2700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b240ffd20 0x7f8b2419a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b2a2a2700 1 -- 172.21.15.78:0/26879894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b200095f0 con 0x7f8b241006b0 2023-07-14T19:02:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b292a0700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b240ffd20 0x7f8b2419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b2a2a2700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b241006b0 0x7f8b2419cba0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c007ee0 tx=0x7f8b1c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b1affd700 1 -- 172.21.15.78:0/26879894 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c017070 con 0x7f8b241006b0 2023-07-14T19:02:52.680 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b1affd700 1 -- 172.21.15.78:0/26879894 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8b1c00f040 con 0x7f8b241006b0 2023-07-14T19:02:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2419d950 con 0x7f8b241006b0 2023-07-14T19:02:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b1affd700 1 -- 172.21.15.78:0/26879894 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c005710 con 0x7f8b241006b0 2023-07-14T19:02:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.672+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2419de60 con 0x7f8b241006b0 2023-07-14T19:02:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.673+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b24105a60 con 0x7f8b241006b0 2023-07-14T19:02:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.674+0000 7f8b1affd700 1 -- 172.21.15.78:0/26879894 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8b1c005870 con 0x7f8b241006b0 2023-07-14T19:02:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.674+0000 7f8b1affd700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b100a12a0 0x7f8b100a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.674+0000 7f8b29aa1700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b100a12a0 0x7f8b100a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.674+0000 7f8b1affd700 1 -- 172.21.15.78:0/26879894 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8b1c0cc9e0 con 0x7f8b241006b0 2023-07-14T19:02:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.675+0000 7f8b29aa1700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b100a12a0 0x7f8b100a3750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8b240ffb80 tx=0x7f8b1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.678+0000 7f8b1affd700 1 -- 172.21.15.78:0/26879894 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8b1c095de0 con 0x7f8b241006b0 2023-07-14T19:02:52.868 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.865+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8b2402c8e0 con 0x7f8b241006b0 2023-07-14T19:02:52.870 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.866+0000 7f8b1affd700 1 -- 172.21.15.78:0/26879894 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7f8b1c099400 con 0x7f8b241006b0 2023-07-14T19:02:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:52.871 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:02:52.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b100a12a0 msgr2=0x7f8b100a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b100a12a0 0x7f8b100a3750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8b240ffb80 tx=0x7f8b1400b410 comp rx=0 tx=0).stop 2023-07-14T19:02:52.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b241006b0 msgr2=0x7f8b2419cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:52.873 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b241006b0 0x7f8b2419cba0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c007ee0 tx=0x7f8b1c00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 shutdown_connections 2023-07-14T19:02:52.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8b100a12a0 0x7f8b100a3750 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8b241006b0 0x7f8b2419cba0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b240feb30 0x7f8b24069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 --2- 172.21.15.78:0/26879894 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8b240ffd20 0x7f8b2419a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:52.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 >> 172.21.15.78:0/26879894 conn(0x7f8b240fa150 msgr2=0x7f8b240fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:52.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 shutdown_connections 2023-07-14T19:02:52.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:52.869+0000 7f8b2bd05700 1 -- 172.21.15.78:0/26879894 wait complete. 2023-07-14T19:02:52.885 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-07-14T19:02:52.885 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T19:02:53.007 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 -- 172.21.15.78:0/1846694512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 msgr2=0x7f7428106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.008 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 --2- 172.21.15.78:0/1846694512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 0x7f7428106f60 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f742000b0a0 tx=0x7f742000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 -- 172.21.15.78:0/1846694512 shutdown_connections 2023-07-14T19:02:53.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 --2- 172.21.15.78:0/1846694512 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74280feb20 0x7f74280fef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 --2- 172.21.15.78:0/1846694512 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 0x7f7428106f60 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 --2- 172.21.15.78:0/1846694512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74280ffd10 0x7f7428100160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 -- 172.21.15.78:0/1846694512 >> 172.21.15.78:0/1846694512 conn(0x7f74280fa100 msgr2=0x7f74280fc570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:53.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 -- 172.21.15.78:0/1846694512 shutdown_connections 2023-07-14T19:02:53.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.002+0000 7f742f720700 1 -- 172.21.15.78:0/1846694512 wait complete. 2023-07-14T19:02:53.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742f720700 1 Processor -- start 2023-07-14T19:02:53.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742f720700 1 -- start start 2023-07-14T19:02:53.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742f720700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74280feb20 0x7f7428102ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:53.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742f720700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74280ffd10 0x7f74281010f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:53.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742f720700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 0x7f7428101630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:53.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742f720700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7428101dd0 con 0x7f74281006a0 2023-07-14T19:02:53.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742f720700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7428101f40 con 0x7f74280ffd10 2023-07-14T19:02:53.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742f720700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f742819f8f0 con 0x7f74280feb20 2023-07-14T19:02:53.012 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742dcbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 0x7f7428101630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:53.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742dcbd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 0x7f7428101630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35034/0 (socket says 172.21.15.78:35034) 2023-07-14T19:02:53.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742dcbd700 1 -- 172.21.15.78:0/910320703 learned_addr learned my addr 172.21.15.78:0/910320703 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:53.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742ccbb700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74280ffd10 0x7f74281010f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:53.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742d4bc700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74280feb20 0x7f7428102ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:53.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742dcbd700 1 -- 172.21.15.78:0/910320703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74280feb20 msgr2=0x7f7428102ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742dcbd700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74280feb20 0x7f7428102ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742dcbd700 1 -- 172.21.15.78:0/910320703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74280ffd10 msgr2=0x7f74281010f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742dcbd700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74280ffd10 0x7f74281010f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742dcbd700 1 -- 172.21.15.78:0/910320703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7420009d00 con 0x7f74281006a0 2023-07-14T19:02:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742ccbb700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74280ffd10 0x7f74281010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.003+0000 7f742d4bc700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74280feb20 0x7f7428102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.004+0000 7f742dcbd700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 0x7f7428101630 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f742000b830 tx=0x7f7420004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.004+0000 7f741a7fc700 1 -- 172.21.15.78:0/910320703 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742000e040 con 0x7f74281006a0 2023-07-14T19:02:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.004+0000 7f741a7fc700 1 -- 172.21.15.78:0/910320703 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7420009440 con 0x7f74281006a0 2023-07-14T19:02:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.004+0000 7f741a7fc700 1 -- 172.21.15.78:0/910320703 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f742001dbe0 con 0x7f74281006a0 2023-07-14T19:02:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.004+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742819fb70 con 0x7f74281006a0 2023-07-14T19:02:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.004+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f742819ff40 con 0x7f74281006a0 2023-07-14T19:02:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.005+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f742810a220 con 0x7f74281006a0 2023-07-14T19:02:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.006+0000 7f741a7fc700 1 -- 172.21.15.78:0/910320703 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7420005450 con 0x7f74281006a0 2023-07-14T19:02:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.006+0000 7f741a7fc700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74140a1220 0x7f74140a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.006+0000 7f741a7fc700 1 -- 172.21.15.78:0/910320703 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f74200cea90 con 0x7f74281006a0 2023-07-14T19:02:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.006+0000 7f742d4bc700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74140a1220 0x7f74140a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.007+0000 7f742d4bc700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74140a1220 0x7f74140a36d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7424005fd0 tx=0x7f7424005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.010+0000 7f741a7fc700 1 -- 172.21.15.78:0/910320703 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7420097f20 con 0x7f74281006a0 2023-07-14T19:02:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.193+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7428068fa0 con 0x7f74140a1220 2023-07-14T19:02:53.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.206+0000 7f741a7fc700 1 -- 172.21.15.78:0/910320703 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418030 (secure 0 0 0) 0x7f7428068fa0 con 0x7f74140a1220 2023-07-14T19:02:53.216 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:53.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74140a1220 msgr2=0x7f74140a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74140a1220 0x7f74140a36d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7424005fd0 tx=0x7f7424005e40 comp rx=0 tx=0).stop 2023-07-14T19:02:53.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 msgr2=0x7f7428101630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 0x7f7428101630 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f742000b830 tx=0x7f7420004d90 comp rx=0 tx=0).stop 2023-07-14T19:02:53.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 shutdown_connections 2023-07-14T19:02:53.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f74140a1220 0x7f74140a36d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f74281006a0 0x7f7428101630 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f74280feb20 0x7f7428102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 --2- 172.21.15.78:0/910320703 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f74280ffd10 0x7f74281010f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 >> 172.21.15.78:0/910320703 conn(0x7f74280fa100 msgr2=0x7f7428068890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:53.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 shutdown_connections 2023-07-14T19:02:53.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.210+0000 7f742f720700 1 -- 172.21.15.78:0/910320703 wait complete. 2023-07-14T19:02:53.221 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-14T19:02:53.226 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2023-07-14T19:02:52.114207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597817,"num_objects":47,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":147,"num_read_kb":126,"num_write":275,"num_write_kb":2981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337004,"kb_used_data":12796,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604820,"statfs":{"total":1151940427776,"available":1151595335680,"internally_reserved":0,"allocated":13103104,"data_stored":7228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5087,"num_objects":23,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":49,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003087"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033648+0000","last_change":"2023-07-14T19:02:30.582427+0000","last_active":"2023-07-14T19:02:36.033648+0000","last_peered":"2023-07-14T19:02:36.033648+0000","last_clean":"2023-07-14T19:02:36.033648+0000","last_became_active":"2023-07-14T19:02:30.582149+0000","last_became_peered":"2023-07-14T19:02:30.582149+0000","last_unstale":"2023-07-14T19:02:36.033648+0000","last_undegraded":"2023-07-14T19:02:36.033648+0000","last_fullsized":"2023-07-14T19:02:36.033648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:30:27.437262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032578+0000","last_change":"2023-07-14T19:02:30.567053+0000","last_active":"2023-07-14T19:02:36.032578+0000","last_peered":"2023-07-14T19:02:36.032578+0000","last_clean":"2023-07-14T19:02:36.032578+0000","last_became_active":"2023-07-14T19:02:30.566736+0000","last_became_peered":"2023-07-14T19:02:30.566736+0000","last_unstale":"2023-07-14T19:02:36.032578+0000","last_undegraded":"2023-07-14T19:02:36.032578+0000","last_fullsized":"2023-07-14T19:02:36.032578+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:22:41.188941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032266+0000","last_change":"2023-07-14T19:02:30.577632+0000","last_active":"2023-07-14T19:02:36.032266+0000","last_peered":"2023-07-14T19:02:36.032266+0000","last_clean":"2023-07-14T19:02:36.032266+0000","last_became_active":"2023-07-14T19:02:30.577300+0000","last_became_peered":"2023-07-14T19:02:30.577300+0000","last_unstale":"2023-07-14T19:02:36.032266+0000","last_undegraded":"2023-07-14T19:02:36.032266+0000","last_fullsized":"2023-07-14T19:02:36.032266+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:49:54.069567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033576+0000","last_change":"2023-07-14T19:02:30.581405+0000","last_active":"2023-07-14T19:02:36.033576+0000","last_peered":"2023-07-14T19:02:36.033576+0000","last_clean":"2023-07-14T19:02:36.033576+0000","last_became_active":"2023-07-14T19:02:30.581113+0000","last_became_peered":"2023-07-14T19:02:30.581113+0000","last_unstale":"2023-07-14T19:02:36.033576+0000","last_undegraded":"2023-07-14T19:02:36.033576+0000","last_fullsized":"2023-07-14T19:02:36.033576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:56:31.168060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032458+0000","last_change":"2023-07-14T19:02:30.574285+0000","last_active":"2023-07-14T19:02:36.032458+0000","last_peered":"2023-07-14T19:02:36.032458+0000","last_clean":"2023-07-14T19:02:36.032458+0000","last_became_active":"2023-07-14T19:02:30.573967+0000","last_became_peered":"2023-07-14T19:02:30.573967+0000","last_unstale":"2023-07-14T19:02:36.032458+0000","last_undegraded":"2023-07-14T19:02:36.032458+0000","last_fullsized":"2023-07-14T19:02:36.032458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:05.555425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034390+0000","last_change":"2023-07-14T19:02:30.574135+0000","last_active":"2023-07-14T19:02:36.034390+0000","last_peered":"2023-07-14T19:02:36.034390+0000","last_clean":"2023-07-14T19:02:36.034390+0000","last_became_active":"2023-07-14T19:02:30.573821+0000","last_became_peered":"2023-07-14T19:02:30.573821+0000","last_unstale":"2023-07-14T19:02:36.034390+0000","last_undegraded":"2023-07-14T19:02:36.034390+0000","last_fullsized":"2023-07-14T19:02:36.034390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:35:07.847570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033210+0000","last_change":"2023-07-14T19:02:30.578051+0000","last_active":"2023-07-14T19:02:36.033210+0000","last_peered":"2023-07-14T19:02:36.033210+0000","last_clean":"2023-07-14T19:02:36.033210+0000","last_became_active":"2023-07-14T19:02:30.577700+0000","last_became_peered":"2023-07-14T19:02:30.577700+0000","last_unstale":"2023-07-14T19:02:36.033210+0000","last_undegraded":"2023-07-14T19:02:36.033210+0000","last_fullsized":"2023-07-14T19:02:36.033210+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:30.607677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811504+0000","last_change":"2023-07-14T19:02:30.567480+0000","last_active":"2023-07-14T19:02:35.811504+0000","last_peered":"2023-07-14T19:02:35.811504+0000","last_clean":"2023-07-14T19:02:35.811504+0000","last_became_active":"2023-07-14T19:02:30.567147+0000","last_became_peered":"2023-07-14T19:02:30.567147+0000","last_unstale":"2023-07-14T19:02:35.811504+0000","last_undegraded":"2023-07-14T19:02:35.811504+0000","last_fullsized":"2023-07-14T19:02:35.811504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:27:53.531353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607383+0000","last_change":"2023-07-14T19:02:30.581936+0000","last_active":"2023-07-14T19:02:35.607383+0000","last_peered":"2023-07-14T19:02:35.607383+0000","last_clean":"2023-07-14T19:02:35.607383+0000","last_became_active":"2023-07-14T19:02:30.579619+0000","last_became_peered":"2023-07-14T19:02:30.579619+0000","last_unstale":"2023-07-14T19:02:35.607383+0000","last_undegraded":"2023-07-14T19:02:35.607383+0000","last_fullsized":"2023-07-14T19:02:35.607383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:33:46.396910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034647+0000","last_change":"2023-07-14T19:02:30.574975+0000","last_active":"2023-07-14T19:02:36.034647+0000","last_peered":"2023-07-14T19:02:36.034647+0000","last_clean":"2023-07-14T19:02:36.034647+0000","last_became_active":"2023-07-14T19:02:30.574633+0000","last_became_peered":"2023-07-14T19:02:30.574633+0000","last_unstale":"2023-07-14T19:02:36.034647+0000","last_undegraded":"2023-07-14T19:02:36.034647+0000","last_fullsized":"2023-07-14T19:02:36.034647+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:59:33.710502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033786+0000","last_change":"2023-07-14T19:02:30.583662+0000","last_active":"2023-07-14T19:02:36.033786+0000","last_peered":"2023-07-14T19:02:36.033786+0000","last_clean":"2023-07-14T19:02:36.033786+0000","last_became_active":"2023-07-14T19:02:30.583352+0000","last_became_peered":"2023-07-14T19:02:30.583352+0000","last_unstale":"2023-07-14T19:02:36.033786+0000","last_undegraded":"2023-07-14T19:02:36.033786+0000","last_fullsized":"2023-07-14T19:02:36.033786+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:10.032122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033481+0000","last_change":"2023-07-14T19:02:30.567685+0000","last_active":"2023-07-14T19:02:36.033481+0000","last_peered":"2023-07-14T19:02:36.033481+0000","last_clean":"2023-07-14T19:02:36.033481+0000","last_became_active":"2023-07-14T19:02:30.566674+0000","last_became_peered":"2023-07-14T19:02:30.566674+0000","last_unstale":"2023-07-14T19:02:36.033481+0000","last_undegraded":"2023-07-14T19:02:36.033481+0000","last_fullsized":"2023-07-14T19:02:36.033481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:54.785930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608702+0000","last_change":"2023-07-14T19:02:30.558012+0000","last_active":"2023-07-14T19:02:35.608702+0000","last_peered":"2023-07-14T19:02:35.608702+0000","last_clean":"2023-07-14T19:02:35.608702+0000","last_became_active":"2023-07-14T19:02:30.557675+0000","last_became_peered":"2023-07-14T19:02:30.557675+0000","last_unstale":"2023-07-14T19:02:35.608702+0000","last_undegraded":"2023-07-14T19:02:35.608702+0000","last_fullsized":"2023-07-14T19:02:35.608702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:56.276352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.031964+0000","last_change":"2023-07-14T19:02:30.570341+0000","last_active":"2023-07-14T19:02:36.031964+0000","last_peered":"2023-07-14T19:02:36.031964+0000","last_clean":"2023-07-14T19:02:36.031964+0000","last_became_active":"2023-07-14T19:02:30.570091+0000","last_became_peered":"2023-07-14T19:02:30.570091+0000","last_unstale":"2023-07-14T19:02:36.031964+0000","last_undegraded":"2023-07-14T19:02:36.031964+0000","last_fullsized":"2023-07-14T19:02:36.031964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:25.486081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032932+0000","last_change":"2023-07-14T19:02:30.573746+0000","last_active":"2023-07-14T19:02:36.032932+0000","last_peered":"2023-07-14T19:02:36.032932+0000","last_clean":"2023-07-14T19:02:36.032932+0000","last_became_active":"2023-07-14T19:02:30.573307+0000","last_became_peered":"2023-07-14T19:02:30.573307+0000","last_unstale":"2023-07-14T19:02:36.032932+0000","last_undegraded":"2023-07-14T19:02:36.032932+0000","last_fullsized":"2023-07-14T19:02:36.032932+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:45.264317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.606816+0000","last_change":"2023-07-14T19:02:30.578352+0000","last_active":"2023-07-14T19:02:35.606816+0000","last_peered":"2023-07-14T19:02:35.606816+0000","last_clean":"2023-07-14T19:02:35.606816+0000","last_became_active":"2023-07-14T19:02:30.577631+0000","last_became_peered":"2023-07-14T19:02:30.577631+0000","last_unstale":"2023-07-14T19:02:35.606816+0000","last_undegraded":"2023-07-14T19:02:35.606816+0000","last_fullsized":"2023-07-14T19:02:35.606816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:38.641986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033532+0000","last_change":"2023-07-14T19:02:30.577813+0000","last_active":"2023-07-14T19:02:36.033532+0000","last_peered":"2023-07-14T19:02:36.033532+0000","last_clean":"2023-07-14T19:02:36.033532+0000","last_became_active":"2023-07-14T19:02:30.577482+0000","last_became_peered":"2023-07-14T19:02:30.577482+0000","last_unstale":"2023-07-14T19:02:36.033532+0000","last_undegraded":"2023-07-14T19:02:36.033532+0000","last_fullsized":"2023-07-14T19:02:36.033532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:17.344131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032871+0000","last_change":"2023-07-14T19:02:30.583353+0000","last_active":"2023-07-14T19:02:36.032871+0000","last_peered":"2023-07-14T19:02:36.032871+0000","last_clean":"2023-07-14T19:02:36.032871+0000","last_became_active":"2023-07-14T19:02:30.583037+0000","last_became_peered":"2023-07-14T19:02:30.583037+0000","last_unstale":"2023-07-14T19:02:36.032871+0000","last_undegraded":"2023-07-14T19:02:36.032871+0000","last_fullsized":"2023-07-14T19:02:36.032871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:52:50.245739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609345+0000","last_change":"2023-07-14T19:02:30.569788+0000","last_active":"2023-07-14T19:02:35.609345+0000","last_peered":"2023-07-14T19:02:35.609345+0000","last_clean":"2023-07-14T19:02:35.609345+0000","last_became_active":"2023-07-14T19:02:30.569461+0000","last_became_peered":"2023-07-14T19:02:30.569461+0000","last_unstale":"2023-07-14T19:02:35.609345+0000","last_undegraded":"2023-07-14T19:02:35.609345+0000","last_fullsized":"2023-07-14T19:02:35.609345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:35:17.252918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608767+0000","last_change":"2023-07-14T19:02:34.585220+0000","last_active":"2023-07-14T19:02:35.608767+0000","last_peered":"2023-07-14T19:02:35.608767+0000","last_clean":"2023-07-14T19:02:35.608767+0000","last_became_active":"2023-07-14T19:02:34.584771+0000","last_became_peered":"2023-07-14T19:02:34.584771+0000","last_unstale":"2023-07-14T19:02:35.608767+0000","last_undegraded":"2023-07-14T19:02:35.608767+0000","last_fullsized":"2023-07-14T19:02:35.608767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:48.195878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"80'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032869+0000","last_change":"2023-07-14T19:02:29.938571+0000","last_active":"2023-07-14T19:02:36.032869+0000","last_peered":"2023-07-14T19:02:36.032869+0000","last_clean":"2023-07-14T19:02:36.032869+0000","last_became_active":"2023-07-14T19:02:29.938270+0000","last_became_peered":"2023-07-14T19:02:29.938270+0000","last_unstale":"2023-07-14T19:02:36.032869+0000","last_undegraded":"2023-07-14T19:02:36.032869+0000","last_fullsized":"2023-07-14T19:02:36.032869+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:24:05.121677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811982+0000","last_change":"2023-07-14T19:02:30.575894+0000","last_active":"2023-07-14T19:02:35.811982+0000","last_peered":"2023-07-14T19:02:35.811982+0000","last_clean":"2023-07-14T19:02:35.811982+0000","last_became_active":"2023-07-14T19:02:30.575607+0000","last_became_peered":"2023-07-14T19:02:30.575607+0000","last_unstale":"2023-07-14T19:02:35.811982+0000","last_undegraded":"2023-07-14T19:02:35.811982+0000","last_fullsized":"2023-07-14T19:02:35.811982+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:30.612104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033103+0000","last_change":"2023-07-14T19:02:34.590761+0000","last_active":"2023-07-14T19:02:36.033103+0000","last_peered":"2023-07-14T19:02:36.033103+0000","last_clean":"2023-07-14T19:02:36.033103+0000","last_became_active":"2023-07-14T19:02:34.590304+0000","last_became_peered":"2023-07-14T19:02:34.590304+0000","last_unstale":"2023-07-14T19:02:36.033103+0000","last_undegraded":"2023-07-14T19:02:36.033103+0000","last_fullsized":"2023-07-14T19:02:36.033103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:18:06.211982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"80'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.812082+0000","last_change":"2023-07-14T19:02:29.944136+0000","last_active":"2023-07-14T19:02:35.812082+0000","last_peered":"2023-07-14T19:02:35.812082+0000","last_clean":"2023-07-14T19:02:35.812082+0000","last_became_active":"2023-07-14T19:02:29.943751+0000","last_became_peered":"2023-07-14T19:02:29.943751+0000","last_unstale":"2023-07-14T19:02:35.812082+0000","last_undegraded":"2023-07-14T19:02:35.812082+0000","last_fullsized":"2023-07-14T19:02:35.812082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:34:19.730141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032337+0000","last_change":"2023-07-14T19:02:30.583156+0000","last_active":"2023-07-14T19:02:36.032337+0000","last_peered":"2023-07-14T19:02:36.032337+0000","last_clean":"2023-07-14T19:02:36.032337+0000","last_became_active":"2023-07-14T19:02:30.582794+0000","last_became_peered":"2023-07-14T19:02:30.582794+0000","last_unstale":"2023-07-14T19:02:36.032337+0000","last_undegraded":"2023-07-14T19:02:36.032337+0000","last_fullsized":"2023-07-14T19:02:36.032337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:48:34.691122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033132+0000","last_change":"2023-07-14T19:02:34.590788+0000","last_active":"2023-07-14T19:02:36.033132+0000","last_peered":"2023-07-14T19:02:36.033132+0000","last_clean":"2023-07-14T19:02:36.033132+0000","last_became_active":"2023-07-14T19:02:34.590385+0000","last_became_peered":"2023-07-14T19:02:34.590385+0000","last_unstale":"2023-07-14T19:02:36.033132+0000","last_undegraded":"2023-07-14T19:02:36.033132+0000","last_fullsized":"2023-07-14T19:02:36.033132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:19:22.697138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.208154+0000","last_change":"2023-07-14T19:02:29.938180+0000","last_active":"2023-07-14T19:02:50.208154+0000","last_peered":"2023-07-14T19:02:50.208154+0000","last_clean":"2023-07-14T19:02:50.208154+0000","last_became_active":"2023-07-14T19:02:29.937823+0000","last_became_peered":"2023-07-14T19:02:29.937823+0000","last_unstale":"2023-07-14T19:02:50.208154+0000","last_undegraded":"2023-07-14T19:02:50.208154+0000","last_fullsized":"2023-07-14T19:02:50.208154+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:08:22.578643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607182+0000","last_change":"2023-07-14T19:02:30.582746+0000","last_active":"2023-07-14T19:02:35.607182+0000","last_peered":"2023-07-14T19:02:35.607182+0000","last_clean":"2023-07-14T19:02:35.607182+0000","last_became_active":"2023-07-14T19:02:30.582426+0000","last_became_peered":"2023-07-14T19:02:30.582426+0000","last_unstale":"2023-07-14T19:02:35.607182+0000","last_undegraded":"2023-07-14T19:02:35.607182+0000","last_fullsized":"2023-07-14T19:02:35.607182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:40.875764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034368+0000","last_change":"2023-07-14T19:02:34.589966+0000","last_active":"2023-07-14T19:02:36.034368+0000","last_peered":"2023-07-14T19:02:36.034368+0000","last_clean":"2023-07-14T19:02:36.034368+0000","last_became_active":"2023-07-14T19:02:34.589544+0000","last_became_peered":"2023-07-14T19:02:34.589544+0000","last_unstale":"2023-07-14T19:02:36.034368+0000","last_undegraded":"2023-07-14T19:02:36.034368+0000","last_fullsized":"2023-07-14T19:02:36.034368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:02:51.191966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032844+0000","last_change":"2023-07-14T19:02:29.935507+0000","last_active":"2023-07-14T19:02:36.032844+0000","last_peered":"2023-07-14T19:02:36.032844+0000","last_clean":"2023-07-14T19:02:36.032844+0000","last_became_active":"2023-07-14T19:02:29.935160+0000","last_became_peered":"2023-07-14T19:02:29.935160+0000","last_unstale":"2023-07-14T19:02:36.032844+0000","last_undegraded":"2023-07-14T19:02:36.032844+0000","last_fullsized":"2023-07-14T19:02:36.032844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:34:50.545076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034296+0000","last_change":"2023-07-14T19:02:30.578077+0000","last_active":"2023-07-14T19:02:36.034296+0000","last_peered":"2023-07-14T19:02:36.034296+0000","last_clean":"2023-07-14T19:02:36.034296+0000","last_became_active":"2023-07-14T19:02:30.577764+0000","last_became_peered":"2023-07-14T19:02:30.577764+0000","last_unstale":"2023-07-14T19:02:36.034296+0000","last_undegraded":"2023-07-14T19:02:36.034296+0000","last_fullsized":"2023-07-14T19:02:36.034296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:20.545387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609074+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:02:35.609074+0000","last_peered":"2023-07-14T19:02:35.609074+0000","last_clean":"2023-07-14T19:02:35.609074+0000","last_became_active":"2023-07-14T19:02:34.583408+0000","last_became_peered":"2023-07-14T19:02:34.583408+0000","last_unstale":"2023-07-14T19:02:35.609074+0000","last_undegraded":"2023-07-14T19:02:35.609074+0000","last_fullsized":"2023-07-14T19:02:35.609074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:56:32.640321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:51.618467+0000","last_change":"2023-07-14T19:02:29.943899+0000","last_active":"2023-07-14T19:02:51.618467+0000","last_peered":"2023-07-14T19:02:51.618467+0000","last_clean":"2023-07-14T19:02:51.618467+0000","last_became_active":"2023-07-14T19:02:29.943585+0000","last_became_peered":"2023-07-14T19:02:29.943585+0000","last_unstale":"2023-07-14T19:02:51.618467+0000","last_undegraded":"2023-07-14T19:02:51.618467+0000","last_fullsized":"2023-07-14T19:02:51.618467+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:29.437187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033528+0000","last_change":"2023-07-14T19:02:30.567147+0000","last_active":"2023-07-14T19:02:36.033528+0000","last_peered":"2023-07-14T19:02:36.033528+0000","last_clean":"2023-07-14T19:02:36.033528+0000","last_became_active":"2023-07-14T19:02:30.566792+0000","last_became_peered":"2023-07-14T19:02:30.566792+0000","last_unstale":"2023-07-14T19:02:36.033528+0000","last_undegraded":"2023-07-14T19:02:36.033528+0000","last_fullsized":"2023-07-14T19:02:36.033528+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:15.802329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609090+0000","last_change":"2023-07-14T19:02:34.579561+0000","last_active":"2023-07-14T19:02:35.609090+0000","last_peered":"2023-07-14T19:02:35.609090+0000","last_clean":"2023-07-14T19:02:35.609090+0000","last_became_active":"2023-07-14T19:02:34.579099+0000","last_became_peered":"2023-07-14T19:02:34.579099+0000","last_unstale":"2023-07-14T19:02:35.609090+0000","last_undegraded":"2023-07-14T19:02:35.609090+0000","last_fullsized":"2023-07-14T19:02:35.609090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:24:00.439199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"80'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034320+0000","last_change":"2023-07-14T19:02:29.936651+0000","last_active":"2023-07-14T19:02:36.034320+0000","last_peered":"2023-07-14T19:02:36.034320+0000","last_clean":"2023-07-14T19:02:36.034320+0000","last_became_active":"2023-07-14T19:02:29.936280+0000","last_became_peered":"2023-07-14T19:02:29.936280+0000","last_unstale":"2023-07-14T19:02:36.034320+0000","last_undegraded":"2023-07-14T19:02:36.034320+0000","last_fullsized":"2023-07-14T19:02:36.034320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:28.735813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033492+0000","last_change":"2023-07-14T19:02:30.575925+0000","last_active":"2023-07-14T19:02:36.033492+0000","last_peered":"2023-07-14T19:02:36.033492+0000","last_clean":"2023-07-14T19:02:36.033492+0000","last_became_active":"2023-07-14T19:02:30.575583+0000","last_became_peered":"2023-07-14T19:02:30.575583+0000","last_unstale":"2023-07-14T19:02:36.033492+0000","last_undegraded":"2023-07-14T19:02:36.033492+0000","last_fullsized":"2023-07-14T19:02:36.033492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:09:04.236418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033394+0000","last_change":"2023-07-14T19:02:34.587094+0000","last_active":"2023-07-14T19:02:36.033394+0000","last_peered":"2023-07-14T19:02:36.033394+0000","last_clean":"2023-07-14T19:02:36.033394+0000","last_became_active":"2023-07-14T19:02:34.586650+0000","last_became_peered":"2023-07-14T19:02:34.586650+0000","last_unstale":"2023-07-14T19:02:36.033394+0000","last_undegraded":"2023-07-14T19:02:36.033394+0000","last_fullsized":"2023-07-14T19:02:36.033394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:08.985468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608485+0000","last_change":"2023-07-14T19:02:29.934651+0000","last_active":"2023-07-14T19:02:35.608485+0000","last_peered":"2023-07-14T19:02:35.608485+0000","last_clean":"2023-07-14T19:02:35.608485+0000","last_became_active":"2023-07-14T19:02:29.934347+0000","last_became_peered":"2023-07-14T19:02:29.934347+0000","last_unstale":"2023-07-14T19:02:35.608485+0000","last_undegraded":"2023-07-14T19:02:35.608485+0000","last_fullsized":"2023-07-14T19:02:35.608485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:34:23.476531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811237+0000","last_change":"2023-07-14T19:02:30.571680+0000","last_active":"2023-07-14T19:02:35.811237+0000","last_peered":"2023-07-14T19:02:35.811237+0000","last_clean":"2023-07-14T19:02:35.811237+0000","last_became_active":"2023-07-14T19:02:30.571367+0000","last_became_peered":"2023-07-14T19:02:30.571367+0000","last_unstale":"2023-07-14T19:02:35.811237+0000","last_undegraded":"2023-07-14T19:02:35.811237+0000","last_fullsized":"2023-07-14T19:02:35.811237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:43.958053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032544+0000","last_change":"2023-07-14T19:02:34.577642+0000","last_active":"2023-07-14T19:02:36.032544+0000","last_peered":"2023-07-14T19:02:36.032544+0000","last_clean":"2023-07-14T19:02:36.032544+0000","last_became_active":"2023-07-14T19:02:34.577232+0000","last_became_peered":"2023-07-14T19:02:34.577232+0000","last_unstale":"2023-07-14T19:02:36.032544+0000","last_undegraded":"2023-07-14T19:02:36.032544+0000","last_fullsized":"2023-07-14T19:02:36.032544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:41:14.778029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033710+0000","last_change":"2023-07-14T19:02:29.940688+0000","last_active":"2023-07-14T19:02:36.033710+0000","last_peered":"2023-07-14T19:02:36.033710+0000","last_clean":"2023-07-14T19:02:36.033710+0000","last_became_active":"2023-07-14T19:02:29.940444+0000","last_became_peered":"2023-07-14T19:02:29.940444+0000","last_unstale":"2023-07-14T19:02:36.033710+0000","last_undegraded":"2023-07-14T19:02:36.033710+0000","last_fullsized":"2023-07-14T19:02:36.033710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:39:02.644668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032080+0000","last_change":"2023-07-14T19:02:30.568127+0000","last_active":"2023-07-14T19:02:36.032080+0000","last_peered":"2023-07-14T19:02:36.032080+0000","last_clean":"2023-07-14T19:02:36.032080+0000","last_became_active":"2023-07-14T19:02:30.567811+0000","last_became_peered":"2023-07-14T19:02:30.567811+0000","last_unstale":"2023-07-14T19:02:36.032080+0000","last_undegraded":"2023-07-14T19:02:36.032080+0000","last_fullsized":"2023-07-14T19:02:36.032080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:07:11.513097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811843+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:02:35.811843+0000","last_peered":"2023-07-14T19:02:35.811843+0000","last_clean":"2023-07-14T19:02:35.811843+0000","last_became_active":"2023-07-14T19:02:34.583480+0000","last_became_peered":"2023-07-14T19:02:34.583480+0000","last_unstale":"2023-07-14T19:02:35.811843+0000","last_undegraded":"2023-07-14T19:02:35.811843+0000","last_fullsized":"2023-07-14T19:02:35.811843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:27.737212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033996+0000","last_change":"2023-07-14T19:02:29.938459+0000","last_active":"2023-07-14T19:02:36.033996+0000","last_peered":"2023-07-14T19:02:36.033996+0000","last_clean":"2023-07-14T19:02:36.033996+0000","last_became_active":"2023-07-14T19:02:29.938212+0000","last_became_peered":"2023-07-14T19:02:29.938212+0000","last_unstale":"2023-07-14T19:02:36.033996+0000","last_undegraded":"2023-07-14T19:02:36.033996+0000","last_fullsized":"2023-07-14T19:02:36.033996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:52:39.556855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607453+0000","last_change":"2023-07-14T19:02:30.586345+0000","last_active":"2023-07-14T19:02:35.607453+0000","last_peered":"2023-07-14T19:02:35.607453+0000","last_clean":"2023-07-14T19:02:35.607453+0000","last_became_active":"2023-07-14T19:02:30.585949+0000","last_became_peered":"2023-07-14T19:02:30.585949+0000","last_unstale":"2023-07-14T19:02:35.607453+0000","last_undegraded":"2023-07-14T19:02:35.607453+0000","last_fullsized":"2023-07-14T19:02:35.607453+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:34:26.531581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.031989+0000","last_change":"2023-07-14T19:02:34.582922+0000","last_active":"2023-07-14T19:02:36.031989+0000","last_peered":"2023-07-14T19:02:36.031989+0000","last_clean":"2023-07-14T19:02:36.031989+0000","last_became_active":"2023-07-14T19:02:34.582146+0000","last_became_peered":"2023-07-14T19:02:34.582146+0000","last_unstale":"2023-07-14T19:02:36.031989+0000","last_undegraded":"2023-07-14T19:02:36.031989+0000","last_fullsized":"2023-07-14T19:02:36.031989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:10:17.978844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.197064+0000","last_change":"2023-07-14T19:02:29.938215+0000","last_active":"2023-07-14T19:02:49.197064+0000","last_peered":"2023-07-14T19:02:49.197064+0000","last_clean":"2023-07-14T19:02:49.197064+0000","last_became_active":"2023-07-14T19:02:29.937889+0000","last_became_peered":"2023-07-14T19:02:29.937889+0000","last_unstale":"2023-07-14T19:02:49.197064+0000","last_undegraded":"2023-07-14T19:02:49.197064+0000","last_fullsized":"2023-07-14T19:02:49.197064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:17.122842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032213+0000","last_change":"2023-07-14T19:02:30.569622+0000","last_active":"2023-07-14T19:02:36.032213+0000","last_peered":"2023-07-14T19:02:36.032213+0000","last_clean":"2023-07-14T19:02:36.032213+0000","last_became_active":"2023-07-14T19:02:30.569201+0000","last_became_peered":"2023-07-14T19:02:30.569201+0000","last_unstale":"2023-07-14T19:02:36.032213+0000","last_undegraded":"2023-07-14T19:02:36.032213+0000","last_fullsized":"2023-07-14T19:02:36.032213+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:27:36.907930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033572+0000","last_change":"2023-07-14T19:02:34.580978+0000","last_active":"2023-07-14T19:02:36.033572+0000","last_peered":"2023-07-14T19:02:36.033572+0000","last_clean":"2023-07-14T19:02:36.033572+0000","last_became_active":"2023-07-14T19:02:34.580503+0000","last_became_peered":"2023-07-14T19:02:34.580503+0000","last_unstale":"2023-07-14T19:02:36.033572+0000","last_undegraded":"2023-07-14T19:02:36.033572+0000","last_fullsized":"2023-07-14T19:02:36.033572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:03:24.374548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.196704+0000","last_change":"2023-07-14T19:02:29.942400+0000","last_active":"2023-07-14T19:02:49.196704+0000","last_peered":"2023-07-14T19:02:49.196704+0000","last_clean":"2023-07-14T19:02:49.196704+0000","last_became_active":"2023-07-14T19:02:29.942080+0000","last_became_peered":"2023-07-14T19:02:29.942080+0000","last_unstale":"2023-07-14T19:02:49.196704+0000","last_undegraded":"2023-07-14T19:02:49.196704+0000","last_fullsized":"2023-07-14T19:02:49.196704+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:07:08.969267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607297+0000","last_change":"2023-07-14T19:02:30.582869+0000","last_active":"2023-07-14T19:02:35.607297+0000","last_peered":"2023-07-14T19:02:35.607297+0000","last_clean":"2023-07-14T19:02:35.607297+0000","last_became_active":"2023-07-14T19:02:30.582558+0000","last_became_peered":"2023-07-14T19:02:30.582558+0000","last_unstale":"2023-07-14T19:02:35.607297+0000","last_undegraded":"2023-07-14T19:02:35.607297+0000","last_fullsized":"2023-07-14T19:02:35.607297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:56:31.547355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033051+0000","last_change":"2023-07-14T19:02:34.582610+0000","last_active":"2023-07-14T19:02:36.033051+0000","last_peered":"2023-07-14T19:02:36.033051+0000","last_clean":"2023-07-14T19:02:36.033051+0000","last_became_active":"2023-07-14T19:02:34.581961+0000","last_became_peered":"2023-07-14T19:02:34.581961+0000","last_unstale":"2023-07-14T19:02:36.033051+0000","last_undegraded":"2023-07-14T19:02:36.033051+0000","last_fullsized":"2023-07-14T19:02:36.033051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:28:06.554521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'10","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:46.595988+0000","last_change":"2023-07-14T19:02:29.939314+0000","last_active":"2023-07-14T19:02:46.595988+0000","last_peered":"2023-07-14T19:02:46.595988+0000","last_clean":"2023-07-14T19:02:46.595988+0000","last_became_active":"2023-07-14T19:02:29.939000+0000","last_became_peered":"2023-07-14T19:02:29.939000+0000","last_unstale":"2023-07-14T19:02:46.595988+0000","last_undegraded":"2023-07-14T19:02:46.595988+0000","last_fullsized":"2023-07-14T19:02:46.595988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:44:31.404753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033492+0000","last_change":"2023-07-14T19:02:30.572052+0000","last_active":"2023-07-14T19:02:36.033492+0000","last_peered":"2023-07-14T19:02:36.033492+0000","last_clean":"2023-07-14T19:02:36.033492+0000","last_became_active":"2023-07-14T19:02:30.571739+0000","last_became_peered":"2023-07-14T19:02:30.571739+0000","last_unstale":"2023-07-14T19:02:36.033492+0000","last_undegraded":"2023-07-14T19:02:36.033492+0000","last_fullsized":"2023-07-14T19:02:36.033492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:23.810314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034605+0000","last_change":"2023-07-14T19:02:34.576663+0000","last_active":"2023-07-14T19:02:36.034605+0000","last_peered":"2023-07-14T19:02:36.034605+0000","last_clean":"2023-07-14T19:02:36.034605+0000","last_became_active":"2023-07-14T19:02:34.576162+0000","last_became_peered":"2023-07-14T19:02:34.576162+0000","last_unstale":"2023-07-14T19:02:36.034605+0000","last_undegraded":"2023-07-14T19:02:36.034605+0000","last_fullsized":"2023-07-14T19:02:36.034605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:56:21.126418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.196082+0000","last_change":"2023-07-14T19:02:29.938859+0000","last_active":"2023-07-14T19:02:49.196082+0000","last_peered":"2023-07-14T19:02:49.196082+0000","last_clean":"2023-07-14T19:02:49.196082+0000","last_became_active":"2023-07-14T19:02:29.938630+0000","last_became_peered":"2023-07-14T19:02:29.938630+0000","last_unstale":"2023-07-14T19:02:49.196082+0000","last_undegraded":"2023-07-14T19:02:49.196082+0000","last_fullsized":"2023-07-14T19:02:49.196082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:50.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608493+0000","last_change":"2023-07-14T19:02:30.573798+0000","last_active":"2023-07-14T19:02:35.608493+0000","last_peered":"2023-07-14T19:02:35.608493+0000","last_clean":"2023-07-14T19:02:35.608493+0000","last_became_active":"2023-07-14T19:02:30.572568+0000","last_became_peered":"2023-07-14T19:02:30.572568+0000","last_unstale":"2023-07-14T19:02:35.608493+0000","last_undegraded":"2023-07-14T19:02:35.608493+0000","last_fullsized":"2023-07-14T19:02:35.608493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:02.901707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032450+0000","last_change":"2023-07-14T19:02:34.582256+0000","last_active":"2023-07-14T19:02:36.032450+0000","last_peered":"2023-07-14T19:02:36.032450+0000","last_clean":"2023-07-14T19:02:36.032450+0000","last_became_active":"2023-07-14T19:02:34.581799+0000","last_became_peered":"2023-07-14T19:02:34.581799+0000","last_unstale":"2023-07-14T19:02:36.032450+0000","last_undegraded":"2023-07-14T19:02:36.032450+0000","last_fullsized":"2023-07-14T19:02:36.032450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:15:22.336363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.197291+0000","last_change":"2023-07-14T19:02:29.943874+0000","last_active":"2023-07-14T19:02:49.197291+0000","last_peered":"2023-07-14T19:02:49.197291+0000","last_clean":"2023-07-14T19:02:49.197291+0000","last_became_active":"2023-07-14T19:02:29.943596+0000","last_became_peered":"2023-07-14T19:02:29.943596+0000","last_unstale":"2023-07-14T19:02:49.197291+0000","last_undegraded":"2023-07-14T19:02:49.197291+0000","last_fullsized":"2023-07-14T19:02:49.197291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:53:58.657606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032901+0000","last_change":"2023-07-14T19:02:30.572157+0000","last_active":"2023-07-14T19:02:36.032901+0000","last_peered":"2023-07-14T19:02:36.032901+0000","last_clean":"2023-07-14T19:02:36.032901+0000","last_became_active":"2023-07-14T19:02:30.571877+0000","last_became_peered":"2023-07-14T19:02:30.571877+0000","last_unstale":"2023-07-14T19:02:36.032901+0000","last_undegraded":"2023-07-14T19:02:36.032901+0000","last_fullsized":"2023-07-14T19:02:36.032901+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:16:17.711397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.031996+0000","last_change":"2023-07-14T19:02:34.588594+0000","last_active":"2023-07-14T19:02:36.031996+0000","last_peered":"2023-07-14T19:02:36.031996+0000","last_clean":"2023-07-14T19:02:36.031996+0000","last_became_active":"2023-07-14T19:02:34.587763+0000","last_became_peered":"2023-07-14T19:02:34.587763+0000","last_unstale":"2023-07-14T19:02:36.031996+0000","last_undegraded":"2023-07-14T19:02:36.031996+0000","last_fullsized":"2023-07-14T19:02:36.031996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:45.839894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:51.619051+0000","last_change":"2023-07-14T19:02:29.939145+0000","last_active":"2023-07-14T19:02:51.619051+0000","last_peered":"2023-07-14T19:02:51.619051+0000","last_clean":"2023-07-14T19:02:51.619051+0000","last_became_active":"2023-07-14T19:02:29.938816+0000","last_became_peered":"2023-07-14T19:02:29.938816+0000","last_unstale":"2023-07-14T19:02:51.619051+0000","last_undegraded":"2023-07-14T19:02:51.619051+0000","last_fullsized":"2023-07-14T19:02:51.619051+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:27:14.358986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032944+0000","last_change":"2023-07-14T19:02:30.572694+0000","last_active":"2023-07-14T19:02:36.032944+0000","last_peered":"2023-07-14T19:02:36.032944+0000","last_clean":"2023-07-14T19:02:36.032944+0000","last_became_active":"2023-07-14T19:02:30.572391+0000","last_became_peered":"2023-07-14T19:02:30.572391+0000","last_unstale":"2023-07-14T19:02:36.032944+0000","last_undegraded":"2023-07-14T19:02:36.032944+0000","last_fullsized":"2023-07-14T19:02:36.032944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:19:15.638964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609130+0000","last_change":"2023-07-14T19:02:34.582476+0000","last_active":"2023-07-14T19:02:35.609130+0000","last_peered":"2023-07-14T19:02:35.609130+0000","last_clean":"2023-07-14T19:02:35.609130+0000","last_became_active":"2023-07-14T19:02:34.582083+0000","last_became_peered":"2023-07-14T19:02:34.582083+0000","last_unstale":"2023-07-14T19:02:35.609130+0000","last_undegraded":"2023-07-14T19:02:35.609130+0000","last_fullsized":"2023-07-14T19:02:35.609130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:53.732894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.210544+0000","last_change":"2023-07-14T19:02:29.941427+0000","last_active":"2023-07-14T19:02:50.210544+0000","last_peered":"2023-07-14T19:02:50.210544+0000","last_clean":"2023-07-14T19:02:50.210544+0000","last_became_active":"2023-07-14T19:02:29.941103+0000","last_became_peered":"2023-07-14T19:02:29.941103+0000","last_unstale":"2023-07-14T19:02:50.210544+0000","last_undegraded":"2023-07-14T19:02:50.210544+0000","last_fullsized":"2023-07-14T19:02:50.210544+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:57:09.780041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034128+0000","last_change":"2023-07-14T19:02:30.571778+0000","last_active":"2023-07-14T19:02:36.034128+0000","last_peered":"2023-07-14T19:02:36.034128+0000","last_clean":"2023-07-14T19:02:36.034128+0000","last_became_active":"2023-07-14T19:02:30.571478+0000","last_became_peered":"2023-07-14T19:02:30.571478+0000","last_unstale":"2023-07-14T19:02:36.034128+0000","last_undegraded":"2023-07-14T19:02:36.034128+0000","last_fullsized":"2023-07-14T19:02:36.034128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:41:32.785043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034066+0000","last_change":"2023-07-14T19:02:34.584170+0000","last_active":"2023-07-14T19:02:36.034066+0000","last_peered":"2023-07-14T19:02:36.034066+0000","last_clean":"2023-07-14T19:02:36.034066+0000","last_became_active":"2023-07-14T19:02:34.583773+0000","last_became_peered":"2023-07-14T19:02:34.583773+0000","last_unstale":"2023-07-14T19:02:36.034066+0000","last_undegraded":"2023-07-14T19:02:36.034066+0000","last_fullsized":"2023-07-14T19:02:36.034066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:52:47.607092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033133+0000","last_change":"2023-07-14T19:02:29.938386+0000","last_active":"2023-07-14T19:02:36.033133+0000","last_peered":"2023-07-14T19:02:36.033133+0000","last_clean":"2023-07-14T19:02:36.033133+0000","last_became_active":"2023-07-14T19:02:29.937394+0000","last_became_peered":"2023-07-14T19:02:29.937394+0000","last_unstale":"2023-07-14T19:02:36.033133+0000","last_undegraded":"2023-07-14T19:02:36.033133+0000","last_fullsized":"2023-07-14T19:02:36.033133+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:32.087374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033189+0000","last_change":"2023-07-14T19:02:30.572485+0000","last_active":"2023-07-14T19:02:36.033189+0000","last_peered":"2023-07-14T19:02:36.033189+0000","last_clean":"2023-07-14T19:02:36.033189+0000","last_became_active":"2023-07-14T19:02:30.572177+0000","last_became_peered":"2023-07-14T19:02:30.572177+0000","last_unstale":"2023-07-14T19:02:36.033189+0000","last_undegraded":"2023-07-14T19:02:36.033189+0000","last_fullsized":"2023-07-14T19:02:36.033189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:48:11.076249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032840+0000","last_change":"2023-07-14T19:02:34.582340+0000","last_active":"2023-07-14T19:02:36.032840+0000","last_peered":"2023-07-14T19:02:36.032840+0000","last_clean":"2023-07-14T19:02:36.032840+0000","last_became_active":"2023-07-14T19:02:34.581936+0000","last_became_peered":"2023-07-14T19:02:34.581936+0000","last_unstale":"2023-07-14T19:02:36.032840+0000","last_undegraded":"2023-07-14T19:02:36.032840+0000","last_fullsized":"2023-07-14T19:02:36.032840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:12.341537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032203+0000","last_change":"2023-07-14T19:02:29.941404+0000","last_active":"2023-07-14T19:02:36.032203+0000","last_peered":"2023-07-14T19:02:36.032203+0000","last_clean":"2023-07-14T19:02:36.032203+0000","last_became_active":"2023-07-14T19:02:29.941052+0000","last_became_peered":"2023-07-14T19:02:29.941052+0000","last_unstale":"2023-07-14T19:02:36.032203+0000","last_undegraded":"2023-07-14T19:02:36.032203+0000","last_fullsized":"2023-07-14T19:02:36.032203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:50.738003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033689+0000","last_change":"2023-07-14T19:02:30.565671+0000","last_active":"2023-07-14T19:02:36.033689+0000","last_peered":"2023-07-14T19:02:36.033689+0000","last_clean":"2023-07-14T19:02:36.033689+0000","last_became_active":"2023-07-14T19:02:30.565375+0000","last_became_peered":"2023-07-14T19:02:30.565375+0000","last_unstale":"2023-07-14T19:02:36.033689+0000","last_undegraded":"2023-07-14T19:02:36.033689+0000","last_fullsized":"2023-07-14T19:02:36.033689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:00.244922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608815+0000","last_change":"2023-07-14T19:02:34.577692+0000","last_active":"2023-07-14T19:02:35.608815+0000","last_peered":"2023-07-14T19:02:35.608815+0000","last_clean":"2023-07-14T19:02:35.608815+0000","last_became_active":"2023-07-14T19:02:34.577329+0000","last_became_peered":"2023-07-14T19:02:34.577329+0000","last_unstale":"2023-07-14T19:02:35.608815+0000","last_undegraded":"2023-07-14T19:02:35.608815+0000","last_fullsized":"2023-07-14T19:02:35.608815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:42.179550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033959+0000","last_change":"2023-07-14T19:02:29.938330+0000","last_active":"2023-07-14T19:02:36.033959+0000","last_peered":"2023-07-14T19:02:36.033959+0000","last_clean":"2023-07-14T19:02:36.033959+0000","last_became_active":"2023-07-14T19:02:29.938022+0000","last_became_peered":"2023-07-14T19:02:29.938022+0000","last_unstale":"2023-07-14T19:02:36.033959+0000","last_undegraded":"2023-07-14T19:02:36.033959+0000","last_fullsized":"2023-07-14T19:02:36.033959+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:58.341498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033320+0000","last_change":"2023-07-14T19:02:30.569353+0000","last_active":"2023-07-14T19:02:36.033320+0000","last_peered":"2023-07-14T19:02:36.033320+0000","last_clean":"2023-07-14T19:02:36.033320+0000","last_became_active":"2023-07-14T19:02:30.569037+0000","last_became_peered":"2023-07-14T19:02:30.569037+0000","last_unstale":"2023-07-14T19:02:36.033320+0000","last_undegraded":"2023-07-14T19:02:36.033320+0000","last_fullsized":"2023-07-14T19:02:36.033320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:43.690826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609120+0000","last_change":"2023-07-14T19:02:34.580786+0000","last_active":"2023-07-14T19:02:35.609120+0000","last_peered":"2023-07-14T19:02:35.609120+0000","last_clean":"2023-07-14T19:02:35.609120+0000","last_became_active":"2023-07-14T19:02:34.580395+0000","last_became_peered":"2023-07-14T19:02:34.580395+0000","last_unstale":"2023-07-14T19:02:35.609120+0000","last_undegraded":"2023-07-14T19:02:35.609120+0000","last_fullsized":"2023-07-14T19:02:35.609120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:24.810182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034087+0000","last_change":"2023-07-14T19:02:30.574302+0000","last_active":"2023-07-14T19:02:36.034087+0000","last_peered":"2023-07-14T19:02:36.034087+0000","last_clean":"2023-07-14T19:02:36.034087+0000","last_became_active":"2023-07-14T19:02:30.573912+0000","last_became_peered":"2023-07-14T19:02:30.573912+0000","last_unstale":"2023-07-14T19:02:36.034087+0000","last_undegraded":"2023-07-14T19:02:36.034087+0000","last_fullsized":"2023-07-14T19:02:36.034087+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:12:45.492316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608417+0000","last_change":"2023-07-14T19:02:29.942302+0000","last_active":"2023-07-14T19:02:35.608417+0000","last_peered":"2023-07-14T19:02:35.608417+0000","last_clean":"2023-07-14T19:02:35.608417+0000","last_became_active":"2023-07-14T19:02:29.942005+0000","last_became_peered":"2023-07-14T19:02:29.942005+0000","last_unstale":"2023-07-14T19:02:35.608417+0000","last_undegraded":"2023-07-14T19:02:35.608417+0000","last_fullsized":"2023-07-14T19:02:35.608417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:55.061870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033583+0000","last_change":"2023-07-14T19:02:34.578248+0000","last_active":"2023-07-14T19:02:36.033583+0000","last_peered":"2023-07-14T19:02:36.033583+0000","last_clean":"2023-07-14T19:02:36.033583+0000","last_became_active":"2023-07-14T19:02:34.577930+0000","last_became_peered":"2023-07-14T19:02:34.577930+0000","last_unstale":"2023-07-14T19:02:36.033583+0000","last_undegraded":"2023-07-14T19:02:36.033583+0000","last_fullsized":"2023-07-14T19:02:36.033583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:51:49.813630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811402+0000","last_change":"2023-07-14T19:02:30.582892+0000","last_active":"2023-07-14T19:02:35.811402+0000","last_peered":"2023-07-14T19:02:35.811402+0000","last_clean":"2023-07-14T19:02:35.811402+0000","last_became_active":"2023-07-14T19:02:30.582588+0000","last_became_peered":"2023-07-14T19:02:30.582588+0000","last_unstale":"2023-07-14T19:02:35.811402+0000","last_undegraded":"2023-07-14T19:02:35.811402+0000","last_fullsized":"2023-07-14T19:02:35.811402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:00:14.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.613182+0000","last_change":"2023-07-14T19:02:29.935857+0000","last_active":"2023-07-14T19:02:50.613182+0000","last_peered":"2023-07-14T19:02:50.613182+0000","last_clean":"2023-07-14T19:02:50.613182+0000","last_became_active":"2023-07-14T19:02:29.935489+0000","last_became_peered":"2023-07-14T19:02:29.935489+0000","last_unstale":"2023-07-14T19:02:50.613182+0000","last_undegraded":"2023-07-14T19:02:50.613182+0000","last_fullsized":"2023-07-14T19:02:50.613182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:17:38.403976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":336,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032647+0000","last_change":"2023-07-14T19:01:31.967476+0000","last_active":"2023-07-14T19:02:36.032647+0000","last_peered":"2023-07-14T19:02:36.032647+0000","last_clean":"2023-07-14T19:02:36.032647+0000","last_became_active":"2023-07-14T19:01:31.967146+0000","last_became_peered":"2023-07-14T19:01:31.967146+0000","last_unstale":"2023-07-14T19:02:36.032647+0000","last_undegraded":"2023-07-14T19:02:36.032647+0000","last_fullsized":"2023-07-14T19:02:36.032647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_clean_scrub_stamp":"2023-07-14T18:58:40.634620+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:05.217720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034257+0000","last_change":"2023-07-14T19:02:34.571823+0000","last_active":"2023-07-14T19:02:36.034257+0000","last_peered":"2023-07-14T19:02:36.034257+0000","last_clean":"2023-07-14T19:02:36.034257+0000","last_became_active":"2023-07-14T19:02:34.571359+0000","last_became_peered":"2023-07-14T19:02:34.571359+0000","last_unstale":"2023-07-14T19:02:36.034257+0000","last_undegraded":"2023-07-14T19:02:36.034257+0000","last_fullsized":"2023-07-14T19:02:36.034257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:33:12.956631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608304+0000","last_change":"2023-07-14T19:02:30.573736+0000","last_active":"2023-07-14T19:02:35.608304+0000","last_peered":"2023-07-14T19:02:35.608304+0000","last_clean":"2023-07-14T19:02:35.608304+0000","last_became_active":"2023-07-14T19:02:30.573381+0000","last_became_peered":"2023-07-14T19:02:30.573381+0000","last_unstale":"2023-07-14T19:02:35.608304+0000","last_undegraded":"2023-07-14T19:02:35.608304+0000","last_fullsized":"2023-07-14T19:02:35.608304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:21:36.599123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.209654+0000","last_change":"2023-07-14T19:02:29.937660+0000","last_active":"2023-07-14T19:02:50.209654+0000","last_peered":"2023-07-14T19:02:50.209654+0000","last_clean":"2023-07-14T19:02:50.209654+0000","last_became_active":"2023-07-14T19:02:29.937348+0000","last_became_peered":"2023-07-14T19:02:29.937348+0000","last_unstale":"2023-07-14T19:02:50.209654+0000","last_undegraded":"2023-07-14T19:02:50.209654+0000","last_fullsized":"2023-07-14T19:02:50.209654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:35:12.947905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034302+0000","last_change":"2023-07-14T19:02:34.581534+0000","last_active":"2023-07-14T19:02:36.034302+0000","last_peered":"2023-07-14T19:02:36.034302+0000","last_clean":"2023-07-14T19:02:36.034302+0000","last_became_active":"2023-07-14T19:02:34.580998+0000","last_became_peered":"2023-07-14T19:02:34.580998+0000","last_unstale":"2023-07-14T19:02:36.034302+0000","last_undegraded":"2023-07-14T19:02:36.034302+0000","last_fullsized":"2023-07-14T19:02:36.034302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:01:16.545480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.196866+0000","last_change":"2023-07-14T19:02:29.942336+0000","last_active":"2023-07-14T19:02:49.196866+0000","last_peered":"2023-07-14T19:02:49.196866+0000","last_clean":"2023-07-14T19:02:49.196866+0000","last_became_active":"2023-07-14T19:02:29.942033+0000","last_became_peered":"2023-07-14T19:02:29.942033+0000","last_unstale":"2023-07-14T19:02:49.196866+0000","last_undegraded":"2023-07-14T19:02:49.196866+0000","last_fullsized":"2023-07-14T19:02:49.196866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:34.783218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608984+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:02:35.608984+0000","last_peered":"2023-07-14T19:02:35.608984+0000","last_clean":"2023-07-14T19:02:35.608984+0000","last_became_active":"2023-07-14T19:02:30.571129+0000","last_became_peered":"2023-07-14T19:02:30.571129+0000","last_unstale":"2023-07-14T19:02:35.608984+0000","last_undegraded":"2023-07-14T19:02:35.608984+0000","last_fullsized":"2023-07-14T19:02:35.608984+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:17.668104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811239+0000","last_change":"2023-07-14T19:02:34.580740+0000","last_active":"2023-07-14T19:02:35.811239+0000","last_peered":"2023-07-14T19:02:35.811239+0000","last_clean":"2023-07-14T19:02:35.811239+0000","last_became_active":"2023-07-14T19:02:34.580344+0000","last_became_peered":"2023-07-14T19:02:34.580344+0000","last_unstale":"2023-07-14T19:02:35.811239+0000","last_undegraded":"2023-07-14T19:02:35.811239+0000","last_fullsized":"2023-07-14T19:02:35.811239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:26:55.627423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032795+0000","last_change":"2023-07-14T19:02:29.941349+0000","last_active":"2023-07-14T19:02:36.032795+0000","last_peered":"2023-07-14T19:02:36.032795+0000","last_clean":"2023-07-14T19:02:36.032795+0000","last_became_active":"2023-07-14T19:02:29.941039+0000","last_became_peered":"2023-07-14T19:02:29.941039+0000","last_unstale":"2023-07-14T19:02:36.032795+0000","last_undegraded":"2023-07-14T19:02:36.032795+0000","last_fullsized":"2023-07-14T19:02:36.032795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:47:27.500876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811807+0000","last_change":"2023-07-14T19:02:30.577546+0000","last_active":"2023-07-14T19:02:35.811807+0000","last_peered":"2023-07-14T19:02:35.811807+0000","last_clean":"2023-07-14T19:02:35.811807+0000","last_became_active":"2023-07-14T19:02:30.576833+0000","last_became_peered":"2023-07-14T19:02:30.576833+0000","last_unstale":"2023-07-14T19:02:35.811807+0000","last_undegraded":"2023-07-14T19:02:35.811807+0000","last_fullsized":"2023-07-14T19:02:35.811807+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:47.060524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608552+0000","last_change":"2023-07-14T19:02:34.580655+0000","last_active":"2023-07-14T19:02:35.608552+0000","last_peered":"2023-07-14T19:02:35.608552+0000","last_clean":"2023-07-14T19:02:35.608552+0000","last_became_active":"2023-07-14T19:02:34.580186+0000","last_became_peered":"2023-07-14T19:02:34.580186+0000","last_unstale":"2023-07-14T19:02:35.608552+0000","last_undegraded":"2023-07-14T19:02:35.608552+0000","last_fullsized":"2023-07-14T19:02:35.608552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:30.211525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.196728+0000","last_change":"2023-07-14T19:02:29.938343+0000","last_active":"2023-07-14T19:02:49.196728+0000","last_peered":"2023-07-14T19:02:49.196728+0000","last_clean":"2023-07-14T19:02:49.196728+0000","last_became_active":"2023-07-14T19:02:29.938053+0000","last_became_peered":"2023-07-14T19:02:29.938053+0000","last_unstale":"2023-07-14T19:02:49.196728+0000","last_undegraded":"2023-07-14T19:02:49.196728+0000","last_fullsized":"2023-07-14T19:02:49.196728+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:06.212318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032632+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:02:36.032632+0000","last_peered":"2023-07-14T19:02:36.032632+0000","last_clean":"2023-07-14T19:02:36.032632+0000","last_became_active":"2023-07-14T19:02:30.572436+0000","last_became_peered":"2023-07-14T19:02:30.572436+0000","last_unstale":"2023-07-14T19:02:36.032632+0000","last_undegraded":"2023-07-14T19:02:36.032632+0000","last_fullsized":"2023-07-14T19:02:36.032632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:58:47.846002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032889+0000","last_change":"2023-07-14T19:02:34.582900+0000","last_active":"2023-07-14T19:02:36.032889+0000","last_peered":"2023-07-14T19:02:36.032889+0000","last_clean":"2023-07-14T19:02:36.032889+0000","last_became_active":"2023-07-14T19:02:34.582493+0000","last_became_peered":"2023-07-14T19:02:34.582493+0000","last_unstale":"2023-07-14T19:02:36.032889+0000","last_undegraded":"2023-07-14T19:02:36.032889+0000","last_fullsized":"2023-07-14T19:02:36.032889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:53:03.101901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.606993+0000","last_change":"2023-07-14T19:02:29.941819+0000","last_active":"2023-07-14T19:02:35.606993+0000","last_peered":"2023-07-14T19:02:35.606993+0000","last_clean":"2023-07-14T19:02:35.606993+0000","last_became_active":"2023-07-14T19:02:29.941469+0000","last_became_peered":"2023-07-14T19:02:29.941469+0000","last_unstale":"2023-07-14T19:02:35.606993+0000","last_undegraded":"2023-07-14T19:02:35.606993+0000","last_fullsized":"2023-07-14T19:02:35.606993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:27.032813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032246+0000","last_change":"2023-07-14T19:02:30.565125+0000","last_active":"2023-07-14T19:02:36.032246+0000","last_peered":"2023-07-14T19:02:36.032246+0000","last_clean":"2023-07-14T19:02:36.032246+0000","last_became_active":"2023-07-14T19:02:30.564820+0000","last_became_peered":"2023-07-14T19:02:30.564820+0000","last_unstale":"2023-07-14T19:02:36.032246+0000","last_undegraded":"2023-07-14T19:02:36.032246+0000","last_fullsized":"2023-07-14T19:02:36.032246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:29.561726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032950+0000","last_change":"2023-07-14T19:02:34.588273+0000","last_active":"2023-07-14T19:02:36.032950+0000","last_peered":"2023-07-14T19:02:36.032950+0000","last_clean":"2023-07-14T19:02:36.032950+0000","last_became_active":"2023-07-14T19:02:34.587202+0000","last_became_peered":"2023-07-14T19:02:34.587202+0000","last_unstale":"2023-07-14T19:02:36.032950+0000","last_undegraded":"2023-07-14T19:02:36.032950+0000","last_fullsized":"2023-07-14T19:02:36.032950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:24.908235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.195784+0000","last_change":"2023-07-14T19:02:29.938027+0000","last_active":"2023-07-14T19:02:49.195784+0000","last_peered":"2023-07-14T19:02:49.195784+0000","last_clean":"2023-07-14T19:02:49.195784+0000","last_became_active":"2023-07-14T19:02:29.937690+0000","last_became_peered":"2023-07-14T19:02:29.937690+0000","last_unstale":"2023-07-14T19:02:49.195784+0000","last_undegraded":"2023-07-14T19:02:49.195784+0000","last_fullsized":"2023-07-14T19:02:49.195784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:03.527958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032503+0000","last_change":"2023-07-14T19:02:30.576369+0000","last_active":"2023-07-14T19:02:36.032503+0000","last_peered":"2023-07-14T19:02:36.032503+0000","last_clean":"2023-07-14T19:02:36.032503+0000","last_became_active":"2023-07-14T19:02:30.576113+0000","last_became_peered":"2023-07-14T19:02:30.576113+0000","last_unstale":"2023-07-14T19:02:36.032503+0000","last_undegraded":"2023-07-14T19:02:36.032503+0000","last_fullsized":"2023-07-14T19:02:36.032503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:11:30.786937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:37.023177+0000","last_change":"2023-07-14T19:02:34.583724+0000","last_active":"2023-07-14T19:02:37.023177+0000","last_peered":"2023-07-14T19:02:37.023177+0000","last_clean":"2023-07-14T19:02:37.023177+0000","last_became_active":"2023-07-14T19:02:34.583218+0000","last_became_peered":"2023-07-14T19:02:34.583218+0000","last_unstale":"2023-07-14T19:02:37.023177+0000","last_undegraded":"2023-07-14T19:02:37.023177+0000","last_fullsized":"2023-07-14T19:02:37.023177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:49:53.961252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608795+0000","last_change":"2023-07-14T19:02:29.938908+0000","last_active":"2023-07-14T19:02:35.608795+0000","last_peered":"2023-07-14T19:02:35.608795+0000","last_clean":"2023-07-14T19:02:35.608795+0000","last_became_active":"2023-07-14T19:02:29.938650+0000","last_became_peered":"2023-07-14T19:02:29.938650+0000","last_unstale":"2023-07-14T19:02:35.608795+0000","last_undegraded":"2023-07-14T19:02:35.608795+0000","last_fullsized":"2023-07-14T19:02:35.608795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:19.121855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811550+0000","last_change":"2023-07-14T19:02:30.571870+0000","last_active":"2023-07-14T19:02:35.811550+0000","last_peered":"2023-07-14T19:02:35.811550+0000","last_clean":"2023-07-14T19:02:35.811550+0000","last_became_active":"2023-07-14T19:02:30.571521+0000","last_became_peered":"2023-07-14T19:02:30.571521+0000","last_unstale":"2023-07-14T19:02:35.811550+0000","last_undegraded":"2023-07-14T19:02:35.811550+0000","last_fullsized":"2023-07-14T19:02:35.811550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:37.931915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034509+0000","last_change":"2023-07-14T19:02:34.589900+0000","last_active":"2023-07-14T19:02:36.034509+0000","last_peered":"2023-07-14T19:02:36.034509+0000","last_clean":"2023-07-14T19:02:36.034509+0000","last_became_active":"2023-07-14T19:02:34.589481+0000","last_became_peered":"2023-07-14T19:02:34.589481+0000","last_unstale":"2023-07-14T19:02:36.034509+0000","last_undegraded":"2023-07-14T19:02:36.034509+0000","last_fullsized":"2023-07-14T19:02:36.034509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:31:58.819621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.197810+0000","last_change":"2023-07-14T19:02:29.940315+0000","last_active":"2023-07-14T19:02:49.197810+0000","last_peered":"2023-07-14T19:02:49.197810+0000","last_clean":"2023-07-14T19:02:49.197810+0000","last_became_active":"2023-07-14T19:02:29.939991+0000","last_became_peered":"2023-07-14T19:02:29.939991+0000","last_unstale":"2023-07-14T19:02:49.197810+0000","last_undegraded":"2023-07-14T19:02:49.197810+0000","last_fullsized":"2023-07-14T19:02:49.197810+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:46.112090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034041+0000","last_change":"2023-07-14T19:02:30.583959+0000","last_active":"2023-07-14T19:02:36.034041+0000","last_peered":"2023-07-14T19:02:36.034041+0000","last_clean":"2023-07-14T19:02:36.034041+0000","last_became_active":"2023-07-14T19:02:30.583657+0000","last_became_peered":"2023-07-14T19:02:30.583657+0000","last_unstale":"2023-07-14T19:02:36.034041+0000","last_undegraded":"2023-07-14T19:02:36.034041+0000","last_fullsized":"2023-07-14T19:02:36.034041+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:38.543164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.606896+0000","last_change":"2023-07-14T19:02:34.579328+0000","last_active":"2023-07-14T19:02:35.606896+0000","last_peered":"2023-07-14T19:02:35.606896+0000","last_clean":"2023-07-14T19:02:35.606896+0000","last_became_active":"2023-07-14T19:02:34.578904+0000","last_became_peered":"2023-07-14T19:02:34.578904+0000","last_unstale":"2023-07-14T19:02:35.606896+0000","last_undegraded":"2023-07-14T19:02:35.606896+0000","last_fullsized":"2023-07-14T19:02:35.606896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:26:48.233983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609263+0000","last_change":"2023-07-14T19:02:29.940972+0000","last_active":"2023-07-14T19:02:35.609263+0000","last_peered":"2023-07-14T19:02:35.609263+0000","last_clean":"2023-07-14T19:02:35.609263+0000","last_became_active":"2023-07-14T19:02:29.940670+0000","last_became_peered":"2023-07-14T19:02:29.940670+0000","last_unstale":"2023-07-14T19:02:35.609263+0000","last_undegraded":"2023-07-14T19:02:35.609263+0000","last_fullsized":"2023-07-14T19:02:35.609263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:09.182828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033918+0000","last_change":"2023-07-14T19:02:30.583714+0000","last_active":"2023-07-14T19:02:36.033918+0000","last_peered":"2023-07-14T19:02:36.033918+0000","last_clean":"2023-07-14T19:02:36.033918+0000","last_became_active":"2023-07-14T19:02:30.583404+0000","last_became_peered":"2023-07-14T19:02:30.583404+0000","last_unstale":"2023-07-14T19:02:36.033918+0000","last_undegraded":"2023-07-14T19:02:36.033918+0000","last_fullsized":"2023-07-14T19:02:36.033918+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:31.678407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032108+0000","last_change":"2023-07-14T19:02:34.582985+0000","last_active":"2023-07-14T19:02:36.032108+0000","last_peered":"2023-07-14T19:02:36.032108+0000","last_clean":"2023-07-14T19:02:36.032108+0000","last_became_active":"2023-07-14T19:02:34.582606+0000","last_became_peered":"2023-07-14T19:02:34.582606+0000","last_unstale":"2023-07-14T19:02:36.032108+0000","last_undegraded":"2023-07-14T19:02:36.032108+0000","last_fullsized":"2023-07-14T19:02:36.032108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:22:30.700576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034472+0000","last_change":"2023-07-14T19:02:29.944064+0000","last_active":"2023-07-14T19:02:36.034472+0000","last_peered":"2023-07-14T19:02:36.034472+0000","last_clean":"2023-07-14T19:02:36.034472+0000","last_became_active":"2023-07-14T19:02:29.943770+0000","last_became_peered":"2023-07-14T19:02:29.943770+0000","last_unstale":"2023-07-14T19:02:36.034472+0000","last_undegraded":"2023-07-14T19:02:36.034472+0000","last_fullsized":"2023-07-14T19:02:36.034472+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:08:09.335177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034216+0000","last_change":"2023-07-14T19:02:30.580668+0000","last_active":"2023-07-14T19:02:36.034216+0000","last_peered":"2023-07-14T19:02:36.034216+0000","last_clean":"2023-07-14T19:02:36.034216+0000","last_became_active":"2023-07-14T19:02:30.580297+0000","last_became_peered":"2023-07-14T19:02:30.580297+0000","last_unstale":"2023-07-14T19:02:36.034216+0000","last_undegraded":"2023-07-14T19:02:36.034216+0000","last_fullsized":"2023-07-14T19:02:36.034216+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:31:20.673679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033333+0000","last_change":"2023-07-14T19:02:34.578894+0000","last_active":"2023-07-14T19:02:36.033333+0000","last_peered":"2023-07-14T19:02:36.033333+0000","last_clean":"2023-07-14T19:02:36.033333+0000","last_became_active":"2023-07-14T19:02:34.578463+0000","last_became_peered":"2023-07-14T19:02:34.578463+0000","last_unstale":"2023-07-14T19:02:36.033333+0000","last_undegraded":"2023-07-14T19:02:36.033333+0000","last_fullsized":"2023-07-14T19:02:36.033333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:59.507751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609171+0000","last_change":"2023-07-14T19:02:29.938430+0000","last_active":"2023-07-14T19:02:35.609171+0000","last_peered":"2023-07-14T19:02:35.609171+0000","last_clean":"2023-07-14T19:02:35.609171+0000","last_became_active":"2023-07-14T19:02:29.938118+0000","last_became_peered":"2023-07-14T19:02:29.938118+0000","last_unstale":"2023-07-14T19:02:35.609171+0000","last_undegraded":"2023-07-14T19:02:35.609171+0000","last_fullsized":"2023-07-14T19:02:35.609171+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:57:08.670837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033869+0000","last_change":"2023-07-14T19:02:30.575124+0000","last_active":"2023-07-14T19:02:36.033869+0000","last_peered":"2023-07-14T19:02:36.033869+0000","last_clean":"2023-07-14T19:02:36.033869+0000","last_became_active":"2023-07-14T19:02:30.574805+0000","last_became_peered":"2023-07-14T19:02:30.574805+0000","last_unstale":"2023-07-14T19:02:36.033869+0000","last_undegraded":"2023-07-14T19:02:36.033869+0000","last_fullsized":"2023-07-14T19:02:36.033869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:36:50.721965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032645+0000","last_change":"2023-07-14T19:02:30.573897+0000","last_active":"2023-07-14T19:02:36.032645+0000","last_peered":"2023-07-14T19:02:36.032645+0000","last_clean":"2023-07-14T19:02:36.032645+0000","last_became_active":"2023-07-14T19:02:30.573585+0000","last_became_peered":"2023-07-14T19:02:30.573585+0000","last_unstale":"2023-07-14T19:02:36.032645+0000","last_undegraded":"2023-07-14T19:02:36.032645+0000","last_fullsized":"2023-07-14T19:02:36.032645+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:02:19.499089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608862+0000","last_change":"2023-07-14T19:02:30.572390+0000","last_active":"2023-07-14T19:02:35.608862+0000","last_peered":"2023-07-14T19:02:35.608862+0000","last_clean":"2023-07-14T19:02:35.608862+0000","last_became_active":"2023-07-14T19:02:30.572093+0000","last_became_peered":"2023-07-14T19:02:30.572093+0000","last_unstale":"2023-07-14T19:02:35.608862+0000","last_undegraded":"2023-07-14T19:02:35.608862+0000","last_fullsized":"2023-07-14T19:02:35.608862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:55:53.445808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608706+0000","last_change":"2023-07-14T19:02:30.572548+0000","last_active":"2023-07-14T19:02:35.608706+0000","last_peered":"2023-07-14T19:02:35.608706+0000","last_clean":"2023-07-14T19:02:35.608706+0000","last_became_active":"2023-07-14T19:02:30.572260+0000","last_became_peered":"2023-07-14T19:02:30.572260+0000","last_unstale":"2023-07-14T19:02:35.608706+0000","last_undegraded":"2023-07-14T19:02:35.608706+0000","last_fullsized":"2023-07-14T19:02:35.608706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:10.013952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032336+0000","last_change":"2023-07-14T19:02:30.569856+0000","last_active":"2023-07-14T19:02:36.032336+0000","last_peered":"2023-07-14T19:02:36.032336+0000","last_clean":"2023-07-14T19:02:36.032336+0000","last_became_active":"2023-07-14T19:02:30.569546+0000","last_became_peered":"2023-07-14T19:02:30.569546+0000","last_unstale":"2023-07-14T19:02:36.032336+0000","last_undegraded":"2023-07-14T19:02:36.032336+0000","last_fullsized":"2023-07-14T19:02:36.032336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:27.292170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608964+0000","last_change":"2023-07-14T19:02:30.570836+0000","last_active":"2023-07-14T19:02:35.608964+0000","last_peered":"2023-07-14T19:02:35.608964+0000","last_clean":"2023-07-14T19:02:35.608964+0000","last_became_active":"2023-07-14T19:02:30.569675+0000","last_became_peered":"2023-07-14T19:02:30.569675+0000","last_unstale":"2023-07-14T19:02:35.608964+0000","last_undegraded":"2023-07-14T19:02:35.608964+0000","last_fullsized":"2023-07-14T19:02:35.608964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:50.028159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033623+0000","last_change":"2023-07-14T19:02:30.566846+0000","last_active":"2023-07-14T19:02:36.033623+0000","last_peered":"2023-07-14T19:02:36.033623+0000","last_clean":"2023-07-14T19:02:36.033623+0000","last_became_active":"2023-07-14T19:02:30.566525+0000","last_became_peered":"2023-07-14T19:02:30.566525+0000","last_unstale":"2023-07-14T19:02:36.033623+0000","last_undegraded":"2023-07-14T19:02:36.033623+0000","last_fullsized":"2023-07-14T19:02:36.033623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:35:39.931938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609207+0000","last_change":"2023-07-14T19:02:30.579680+0000","last_active":"2023-07-14T19:02:35.609207+0000","last_peered":"2023-07-14T19:02:35.609207+0000","last_clean":"2023-07-14T19:02:35.609207+0000","last_became_active":"2023-07-14T19:02:30.579379+0000","last_became_peered":"2023-07-14T19:02:30.579379+0000","last_unstale":"2023-07-14T19:02:35.609207+0000","last_undegraded":"2023-07-14T19:02:35.609207+0000","last_fullsized":"2023-07-14T19:02:35.609207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:37:18.570317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811962+0000","last_change":"2023-07-14T19:02:30.577722+0000","last_active":"2023-07-14T19:02:35.811962+0000","last_peered":"2023-07-14T19:02:35.811962+0000","last_clean":"2023-07-14T19:02:35.811962+0000","last_became_active":"2023-07-14T19:02:30.577247+0000","last_became_peered":"2023-07-14T19:02:30.577247+0000","last_unstale":"2023-07-14T19:02:35.811962+0000","last_undegraded":"2023-07-14T19:02:35.811962+0000","last_fullsized":"2023-07-14T19:02:35.811962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:29.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033802+0000","last_change":"2023-07-14T19:02:30.571052+0000","last_active":"2023-07-14T19:02:36.033802+0000","last_peered":"2023-07-14T19:02:36.033802+0000","last_clean":"2023-07-14T19:02:36.033802+0000","last_became_active":"2023-07-14T19:02:30.570722+0000","last_became_peered":"2023-07-14T19:02:30.570722+0000","last_unstale":"2023-07-14T19:02:36.033802+0000","last_undegraded":"2023-07-14T19:02:36.033802+0000","last_fullsized":"2023-07-14T19:02:36.033802+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:38.060681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.812013+0000","last_change":"2023-07-14T19:02:30.575685+0000","last_active":"2023-07-14T19:02:35.812013+0000","last_peered":"2023-07-14T19:02:35.812013+0000","last_clean":"2023-07-14T19:02:35.812013+0000","last_became_active":"2023-07-14T19:02:30.575379+0000","last_became_peered":"2023-07-14T19:02:30.575379+0000","last_unstale":"2023-07-14T19:02:35.812013+0000","last_undegraded":"2023-07-14T19:02:35.812013+0000","last_fullsized":"2023-07-14T19:02:35.812013+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:31.763621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607133+0000","last_change":"2023-07-14T19:02:30.582450+0000","last_active":"2023-07-14T19:02:35.607133+0000","last_peered":"2023-07-14T19:02:35.607133+0000","last_clean":"2023-07-14T19:02:35.607133+0000","last_became_active":"2023-07-14T19:02:30.582074+0000","last_became_peered":"2023-07-14T19:02:30.582074+0000","last_unstale":"2023-07-14T19:02:35.607133+0000","last_undegraded":"2023-07-14T19:02:35.607133+0000","last_fullsized":"2023-07-14T19:02:35.607133+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:46.572335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608673+0000","last_change":"2023-07-14T19:02:30.573042+0000","last_active":"2023-07-14T19:02:35.608673+0000","last_peered":"2023-07-14T19:02:35.608673+0000","last_clean":"2023-07-14T19:02:35.608673+0000","last_became_active":"2023-07-14T19:02:30.572498+0000","last_became_peered":"2023-07-14T19:02:30.572498+0000","last_unstale":"2023-07-14T19:02:35.608673+0000","last_undegraded":"2023-07-14T19:02:35.608673+0000","last_fullsized":"2023-07-14T19:02:35.608673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:42:33.413533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033067+0000","last_change":"2023-07-14T19:02:30.569441+0000","last_active":"2023-07-14T19:02:36.033067+0000","last_peered":"2023-07-14T19:02:36.033067+0000","last_clean":"2023-07-14T19:02:36.033067+0000","last_became_active":"2023-07-14T19:02:30.569180+0000","last_became_peered":"2023-07-14T19:02:30.569180+0000","last_unstale":"2023-07-14T19:02:36.033067+0000","last_undegraded":"2023-07-14T19:02:36.033067+0000","last_fullsized":"2023-07-14T19:02:36.033067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:26:58.302404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7326,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":93,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":20258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514511,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":452924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]}]},{"osd":10,"up_from":69,"seq":296352743442,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28528,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716624,"statfs":{"total":95995035648,"available":95965822976,"internally_reserved":0,"allocated":1548288,"data_stored":1047280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.621}]},{"osd":3,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.72499999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.68999999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.65600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":9,"up_from":64,"seq":274877906966,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":453506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39000000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.35999999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51600000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58399999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.62},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.629}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.49299999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135899,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":453331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88300000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60299999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90900000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76300000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72899999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.878,"15min":0.878},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.88500000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.68400000000000005,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.92700000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.626,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.997}]},{"osd":6,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.94299999999999995,"5min":1.01,"15min":1.01},"last":0.90900000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.871,"15min":0.871},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0369999999999999,"5min":1.046,"15min":1.046},"last":0.93999999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:02:11 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":1.036},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.877}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":1.016},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":1.0700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]}]},{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1536000,"data_stored":1044715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39500000000000002,"15min":0.38900000000000001},"min":{"1min":0.28000000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63600000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41099999999999998,"15min":0.40400000000000003},"min":{"1min":0.30299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41599999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.55500000000000005,"15min":0.54300000000000004},"min":{"1min":0.41499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.75800000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.53900000000000003,"15min":0.52900000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.622}]},{"osd":4,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46000000000000002,"15min":0.45300000000000001},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.48499999999999999},"min":{"1min":0.33800000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.44500000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46800000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61099999999999999,"15min":0.60099999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.627},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.65000000000000002,"15min":0.63700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72399999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.66300000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:02:07 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.69399999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.74099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528356,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":452924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.40100000000000002,"15min":0.38700000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.42999999999999999,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.30099999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57899999999999996,"15min":0.55400000000000005},"min":{"1min":0.40200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55300000000000005,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":0.48599999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.59199999999999997},"min":{"1min":0.46300000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.70999999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.5,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":0.40300000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.59699999999999998,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.64300000000000002,"15min":0.623},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65700000000000003,"15min":0.65500000000000003},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:02:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68600000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.76200000000000001,"15min":0.73499999999999999},"min":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.6709999999999998,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.628}]},{"osd":8,"last update":"Fri Jul 14 19:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.62}]},{"osd":9,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.501},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.44600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:02:31 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.66400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":456718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.5,"15min":0.48599999999999999},"min":{"1min":0.39300000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.441},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.377,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66200000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.503}]},{"osd":2,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.55200000000000005,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.52500000000000002,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.67300000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.57099999999999995,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79800000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77400000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:02:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.40500000000000003,"15min":0.377},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.41399999999999998,"15min":0.39200000000000002},"min":{"1min":0.34000000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.69999999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59199999999999997,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.52100000000000002,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.58299999999999996,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68600000000000005}]},{"osd":8,"last update":"Fri Jul 14 19:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.93600000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.83299999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.63500000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":454129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.53100000000000003,"15min":0.503},"min":{"1min":0.371,"5min":0.159,"15min":0.159},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.626},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.502,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.496}]},{"osd":2,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.37,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.746,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58899999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.29499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.33200000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54200000000000004,"15min":0.52300000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.52200000000000002,"15min":0.5},"min":{"1min":0.46400000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60999999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.46000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91700000000000004,"5min":1.042,"15min":1.042},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.41799999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.504,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.58599999999999997,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.67000000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.74199999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70199999999999996,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.024,"15min":1.024},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.66400000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.55600000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.56699999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":457326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78700000000000003,"5min":1.111,"15min":1.111},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.82399999999999995}]},{"osd":1,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.55600000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53900000000000003,"15min":0.53600000000000003},"min":{"1min":0.34699999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.77600000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.752},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.628}]},{"osd":4,"last update":"Fri Jul 14 19:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.69999999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.378},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.433}]},{"osd":6,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.57199999999999995,"15min":0.55800000000000005},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.88},{"interface":"front","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60199999999999998,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61599999999999999,"15min":0.60699999999999998},"min":{"1min":0.47499999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64800000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.59999999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.80600000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.85599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182448,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":453028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.26600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.54800000000000004,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.371,"5min":0.34999999999999998,"15min":0.34200000000000003},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.32400000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.315,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76900000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.41399999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69999999999999996,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.626}]},{"osd":2,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.41499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.82499999999999996,"5min":4.6509999999999998,"15min":4.6509999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63600000000000001,"15min":0.67300000000000004},"min":{"1min":0.41599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79100000000000004,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.56100000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:02:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.53400000000000003,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.39200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.498,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.64000000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.33700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86599999999999999,"5min":1.119,"15min":1.119},"last":0.73999999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76500000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.48999999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.53500000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.69099999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]},{"osd":4,"up_from":33,"seq":141733920813,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":453637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.503,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.38100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.36299999999999999,"15min":0.34100000000000003},"min":{"1min":0.29999999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.90500000000000003,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.377,"15min":0.35699999999999998},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.751,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54700000000000004,"15min":0.53100000000000003},"min":{"1min":0.42599999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56100000000000005,"15min":0.53500000000000003},"min":{"1min":0.40200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.53100000000000003,"15min":0.505},"min":{"1min":0.36099999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93700000000000006,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.52600000000000002,"15min":0.498},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.71799999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57599999999999996,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.751},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.45000000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.625}]},{"osd":7,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.39400000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80500000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.81599999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68600000000000005,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.64900000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73799999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.86199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757288,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1552384,"data_stored":1048875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.5},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.41399999999999998,"15min":0.38900000000000001},"min":{"1min":0.309,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.56200000000000006}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70599999999999996,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.874},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.59599999999999997,"15min":0.58599999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64300000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:02:26 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.89000000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.754},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.90400000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.81799999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.85799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:02:53.237 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-07-14T19:02:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- 172.21.15.78:0/1202007704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80fed40 msgr2=0x7fa2f80ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 --2- 172.21.15.78:0/1202007704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80fed40 0x7fa2f80ff1b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa2e4009990 tx=0x7fa2e4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- 172.21.15.78:0/1202007704 shutdown_connections 2023-07-14T19:02:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 --2- 172.21.15.78:0/1202007704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f80ff6f0 0x7fa2f8106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 --2- 172.21.15.78:0/1202007704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2f8100180 0x7fa2f8100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 --2- 172.21.15.78:0/1202007704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80fed40 0x7fa2f80ff1b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- 172.21.15.78:0/1202007704 >> 172.21.15.78:0/1202007704 conn(0x7fa2f80f98a0 msgr2=0x7fa2f80fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- 172.21.15.78:0/1202007704 shutdown_connections 2023-07-14T19:02:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- 172.21.15.78:0/1202007704 wait complete. 2023-07-14T19:02:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 Processor -- start 2023-07-14T19:02:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- start start 2023-07-14T19:02:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2f80fed40 0x7fa2f81957e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80ff6f0 0x7fa2f8195d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f8100180 0x7fa2f819cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f819d360 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f819d4a0 con 0x7fa2f8100180 2023-07-14T19:02:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2feee0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f819d630 con 0x7fa2f80fed40 2023-07-14T19:02:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2f7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80ff6f0 0x7fa2f8195d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.383+0000 7fa2f7fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80ff6f0 0x7fa2f8195d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35046/0 (socket says 172.21.15.78:35046) 2023-07-14T19:02:53.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f7fff700 1 -- 172.21.15.78:0/274345981 learned_addr learned my addr 172.21.15.78:0/274345981 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:53.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2fcc7c700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2f80fed40 0x7fa2f81957e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:53.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2fd47d700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f8100180 0x7fa2f819cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f7fff700 1 -- 172.21.15.78:0/274345981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2f80fed40 msgr2=0x7fa2f81957e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f7fff700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2f80fed40 0x7fa2f81957e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f7fff700 1 -- 172.21.15.78:0/274345981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f8100180 msgr2=0x7fa2f819cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f7fff700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f8100180 0x7fa2f819cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f7fff700 1 -- 172.21.15.78:0/274345981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e40095f0 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2fd47d700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f8100180 0x7fa2f819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:02:53.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f7fff700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80ff6f0 0x7fa2f8195d20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40098e0 tx=0x7fa2e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f5ffb700 1 -- 172.21.15.78:0/274345981 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e401d070 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f819d8b0 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f5ffb700 1 -- 172.21.15.78:0/274345981 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa2e400be30 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2f5ffb700 1 -- 172.21.15.78:0/274345981 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2e4022e00 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.384+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f819dcd0 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.385+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f810a1d0 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.386+0000 7fa2f5ffb700 1 -- 172.21.15.78:0/274345981 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa2e400b9a0 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.387+0000 7fa2f5ffb700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa2e80a12a0 0x7fa2e80a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.387+0000 7fa2fcc7c700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa2e80a12a0 0x7fa2e80a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.387+0000 7fa2f5ffb700 1 -- 172.21.15.78:0/274345981 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa2e40ce290 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.387+0000 7fa2fcc7c700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa2e80a12a0 0x7fa2e80a3750 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009c60 tx=0x7fa2ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.391+0000 7fa2f5ffb700 1 -- 172.21.15.78:0/274345981 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa2e40977e0 con 0x7fa2f80ff6f0 2023-07-14T19:02:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:53 smithi088 ceph-mon[128973]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:53 smithi088 ceph-mon[128973]: pgmap v266: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2023-07-14T19:02:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:53 smithi088 ceph-mon[128973]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:53.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:53 smithi088 ceph-mon[128973]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:02:53.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:53 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3705707604' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:53.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:53 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/26879894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:53.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:53 smithi078 ceph-mon[127425]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:53.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:53 smithi078 ceph-mon[127425]: pgmap v266: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2023-07-14T19:02:53.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:53 smithi078 ceph-mon[127425]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:53.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:53 smithi078 ceph-mon[127425]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:02:53.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:53 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3705707604' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:53.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:53 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/26879894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:53.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.577+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2f80fe380 con 0x7fa2e80a12a0 2023-07-14T19:02:53.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.588+0000 7fa2f5ffb700 1 -- 172.21.15.78:0/274345981 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+418030 (secure 0 0 0) 0x7fa2f80fe380 con 0x7fa2e80a12a0 2023-07-14T19:02:53.597 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa2e80a12a0 msgr2=0x7fa2e80a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa2e80a12a0 0x7fa2e80a3750 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec009c60 tx=0x7fa2ec009260 comp rx=0 tx=0).stop 2023-07-14T19:02:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80ff6f0 msgr2=0x7fa2f8195d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80ff6f0 0x7fa2f8195d20 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fa2e40098e0 tx=0x7fa2e4004730 comp rx=0 tx=0).stop 2023-07-14T19:02:53.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 shutdown_connections 2023-07-14T19:02:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa2e80a12a0 0x7fa2e80a3750 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa2f8100180 0x7fa2f819cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2f80fed40 0x7fa2f81957e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 --2- 172.21.15.78:0/274345981 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa2f80ff6f0 0x7fa2f8195d20 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 >> 172.21.15.78:0/274345981 conn(0x7fa2f80f98a0 msgr2=0x7fa2f80fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 shutdown_connections 2023-07-14T19:02:53.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:53.592+0000 7fa2feee0700 1 -- 172.21.15.78:0/274345981 wait complete. 2023-07-14T19:02:53.602 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-14T19:02:53.608 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2023-07-14T19:02:52.114207+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":597817,"num_objects":47,"num_object_clones":0,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":0,"num_read":147,"num_read_kb":126,"num_write":275,"num_write_kb":2981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":224,"ondisk_log_size":224,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337004,"kb_used_data":12796,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604820,"statfs":{"total":1151940427776,"available":1151595335680,"internally_reserved":0,"allocated":13103104,"data_stored":7228393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5087,"num_objects":23,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":49,"num_write_kb":29,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003087"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033648+0000","last_change":"2023-07-14T19:02:30.582427+0000","last_active":"2023-07-14T19:02:36.033648+0000","last_peered":"2023-07-14T19:02:36.033648+0000","last_clean":"2023-07-14T19:02:36.033648+0000","last_became_active":"2023-07-14T19:02:30.582149+0000","last_became_peered":"2023-07-14T19:02:30.582149+0000","last_unstale":"2023-07-14T19:02:36.033648+0000","last_undegraded":"2023-07-14T19:02:36.033648+0000","last_fullsized":"2023-07-14T19:02:36.033648+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:30:27.437262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032578+0000","last_change":"2023-07-14T19:02:30.567053+0000","last_active":"2023-07-14T19:02:36.032578+0000","last_peered":"2023-07-14T19:02:36.032578+0000","last_clean":"2023-07-14T19:02:36.032578+0000","last_became_active":"2023-07-14T19:02:30.566736+0000","last_became_peered":"2023-07-14T19:02:30.566736+0000","last_unstale":"2023-07-14T19:02:36.032578+0000","last_undegraded":"2023-07-14T19:02:36.032578+0000","last_fullsized":"2023-07-14T19:02:36.032578+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:22:41.188941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032266+0000","last_change":"2023-07-14T19:02:30.577632+0000","last_active":"2023-07-14T19:02:36.032266+0000","last_peered":"2023-07-14T19:02:36.032266+0000","last_clean":"2023-07-14T19:02:36.032266+0000","last_became_active":"2023-07-14T19:02:30.577300+0000","last_became_peered":"2023-07-14T19:02:30.577300+0000","last_unstale":"2023-07-14T19:02:36.032266+0000","last_undegraded":"2023-07-14T19:02:36.032266+0000","last_fullsized":"2023-07-14T19:02:36.032266+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:49:54.069567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033576+0000","last_change":"2023-07-14T19:02:30.581405+0000","last_active":"2023-07-14T19:02:36.033576+0000","last_peered":"2023-07-14T19:02:36.033576+0000","last_clean":"2023-07-14T19:02:36.033576+0000","last_became_active":"2023-07-14T19:02:30.581113+0000","last_became_peered":"2023-07-14T19:02:30.581113+0000","last_unstale":"2023-07-14T19:02:36.033576+0000","last_undegraded":"2023-07-14T19:02:36.033576+0000","last_fullsized":"2023-07-14T19:02:36.033576+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:56:31.168060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032458+0000","last_change":"2023-07-14T19:02:30.574285+0000","last_active":"2023-07-14T19:02:36.032458+0000","last_peered":"2023-07-14T19:02:36.032458+0000","last_clean":"2023-07-14T19:02:36.032458+0000","last_became_active":"2023-07-14T19:02:30.573967+0000","last_became_peered":"2023-07-14T19:02:30.573967+0000","last_unstale":"2023-07-14T19:02:36.032458+0000","last_undegraded":"2023-07-14T19:02:36.032458+0000","last_fullsized":"2023-07-14T19:02:36.032458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:05.555425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034390+0000","last_change":"2023-07-14T19:02:30.574135+0000","last_active":"2023-07-14T19:02:36.034390+0000","last_peered":"2023-07-14T19:02:36.034390+0000","last_clean":"2023-07-14T19:02:36.034390+0000","last_became_active":"2023-07-14T19:02:30.573821+0000","last_became_peered":"2023-07-14T19:02:30.573821+0000","last_unstale":"2023-07-14T19:02:36.034390+0000","last_undegraded":"2023-07-14T19:02:36.034390+0000","last_fullsized":"2023-07-14T19:02:36.034390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:35:07.847570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033210+0000","last_change":"2023-07-14T19:02:30.578051+0000","last_active":"2023-07-14T19:02:36.033210+0000","last_peered":"2023-07-14T19:02:36.033210+0000","last_clean":"2023-07-14T19:02:36.033210+0000","last_became_active":"2023-07-14T19:02:30.577700+0000","last_became_peered":"2023-07-14T19:02:30.577700+0000","last_unstale":"2023-07-14T19:02:36.033210+0000","last_undegraded":"2023-07-14T19:02:36.033210+0000","last_fullsized":"2023-07-14T19:02:36.033210+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:44:30.607677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811504+0000","last_change":"2023-07-14T19:02:30.567480+0000","last_active":"2023-07-14T19:02:35.811504+0000","last_peered":"2023-07-14T19:02:35.811504+0000","last_clean":"2023-07-14T19:02:35.811504+0000","last_became_active":"2023-07-14T19:02:30.567147+0000","last_became_peered":"2023-07-14T19:02:30.567147+0000","last_unstale":"2023-07-14T19:02:35.811504+0000","last_undegraded":"2023-07-14T19:02:35.811504+0000","last_fullsized":"2023-07-14T19:02:35.811504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:27:53.531353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607383+0000","last_change":"2023-07-14T19:02:30.581936+0000","last_active":"2023-07-14T19:02:35.607383+0000","last_peered":"2023-07-14T19:02:35.607383+0000","last_clean":"2023-07-14T19:02:35.607383+0000","last_became_active":"2023-07-14T19:02:30.579619+0000","last_became_peered":"2023-07-14T19:02:30.579619+0000","last_unstale":"2023-07-14T19:02:35.607383+0000","last_undegraded":"2023-07-14T19:02:35.607383+0000","last_fullsized":"2023-07-14T19:02:35.607383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:33:46.396910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034647+0000","last_change":"2023-07-14T19:02:30.574975+0000","last_active":"2023-07-14T19:02:36.034647+0000","last_peered":"2023-07-14T19:02:36.034647+0000","last_clean":"2023-07-14T19:02:36.034647+0000","last_became_active":"2023-07-14T19:02:30.574633+0000","last_became_peered":"2023-07-14T19:02:30.574633+0000","last_unstale":"2023-07-14T19:02:36.034647+0000","last_undegraded":"2023-07-14T19:02:36.034647+0000","last_fullsized":"2023-07-14T19:02:36.034647+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:59:33.710502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033786+0000","last_change":"2023-07-14T19:02:30.583662+0000","last_active":"2023-07-14T19:02:36.033786+0000","last_peered":"2023-07-14T19:02:36.033786+0000","last_clean":"2023-07-14T19:02:36.033786+0000","last_became_active":"2023-07-14T19:02:30.583352+0000","last_became_peered":"2023-07-14T19:02:30.583352+0000","last_unstale":"2023-07-14T19:02:36.033786+0000","last_undegraded":"2023-07-14T19:02:36.033786+0000","last_fullsized":"2023-07-14T19:02:36.033786+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:26:10.032122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033481+0000","last_change":"2023-07-14T19:02:30.567685+0000","last_active":"2023-07-14T19:02:36.033481+0000","last_peered":"2023-07-14T19:02:36.033481+0000","last_clean":"2023-07-14T19:02:36.033481+0000","last_became_active":"2023-07-14T19:02:30.566674+0000","last_became_peered":"2023-07-14T19:02:30.566674+0000","last_unstale":"2023-07-14T19:02:36.033481+0000","last_undegraded":"2023-07-14T19:02:36.033481+0000","last_fullsized":"2023-07-14T19:02:36.033481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:32:54.785930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608702+0000","last_change":"2023-07-14T19:02:30.558012+0000","last_active":"2023-07-14T19:02:35.608702+0000","last_peered":"2023-07-14T19:02:35.608702+0000","last_clean":"2023-07-14T19:02:35.608702+0000","last_became_active":"2023-07-14T19:02:30.557675+0000","last_became_peered":"2023-07-14T19:02:30.557675+0000","last_unstale":"2023-07-14T19:02:35.608702+0000","last_undegraded":"2023-07-14T19:02:35.608702+0000","last_fullsized":"2023-07-14T19:02:35.608702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:56.276352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.031964+0000","last_change":"2023-07-14T19:02:30.570341+0000","last_active":"2023-07-14T19:02:36.031964+0000","last_peered":"2023-07-14T19:02:36.031964+0000","last_clean":"2023-07-14T19:02:36.031964+0000","last_became_active":"2023-07-14T19:02:30.570091+0000","last_became_peered":"2023-07-14T19:02:30.570091+0000","last_unstale":"2023-07-14T19:02:36.031964+0000","last_undegraded":"2023-07-14T19:02:36.031964+0000","last_fullsized":"2023-07-14T19:02:36.031964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:25.486081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032932+0000","last_change":"2023-07-14T19:02:30.573746+0000","last_active":"2023-07-14T19:02:36.032932+0000","last_peered":"2023-07-14T19:02:36.032932+0000","last_clean":"2023-07-14T19:02:36.032932+0000","last_became_active":"2023-07-14T19:02:30.573307+0000","last_became_peered":"2023-07-14T19:02:30.573307+0000","last_unstale":"2023-07-14T19:02:36.032932+0000","last_undegraded":"2023-07-14T19:02:36.032932+0000","last_fullsized":"2023-07-14T19:02:36.032932+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:45.264317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.606816+0000","last_change":"2023-07-14T19:02:30.578352+0000","last_active":"2023-07-14T19:02:35.606816+0000","last_peered":"2023-07-14T19:02:35.606816+0000","last_clean":"2023-07-14T19:02:35.606816+0000","last_became_active":"2023-07-14T19:02:30.577631+0000","last_became_peered":"2023-07-14T19:02:30.577631+0000","last_unstale":"2023-07-14T19:02:35.606816+0000","last_undegraded":"2023-07-14T19:02:35.606816+0000","last_fullsized":"2023-07-14T19:02:35.606816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:38.641986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033532+0000","last_change":"2023-07-14T19:02:30.577813+0000","last_active":"2023-07-14T19:02:36.033532+0000","last_peered":"2023-07-14T19:02:36.033532+0000","last_clean":"2023-07-14T19:02:36.033532+0000","last_became_active":"2023-07-14T19:02:30.577482+0000","last_became_peered":"2023-07-14T19:02:30.577482+0000","last_unstale":"2023-07-14T19:02:36.033532+0000","last_undegraded":"2023-07-14T19:02:36.033532+0000","last_fullsized":"2023-07-14T19:02:36.033532+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:17.344131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032871+0000","last_change":"2023-07-14T19:02:30.583353+0000","last_active":"2023-07-14T19:02:36.032871+0000","last_peered":"2023-07-14T19:02:36.032871+0000","last_clean":"2023-07-14T19:02:36.032871+0000","last_became_active":"2023-07-14T19:02:30.583037+0000","last_became_peered":"2023-07-14T19:02:30.583037+0000","last_unstale":"2023-07-14T19:02:36.032871+0000","last_undegraded":"2023-07-14T19:02:36.032871+0000","last_fullsized":"2023-07-14T19:02:36.032871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:52:50.245739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609345+0000","last_change":"2023-07-14T19:02:30.569788+0000","last_active":"2023-07-14T19:02:35.609345+0000","last_peered":"2023-07-14T19:02:35.609345+0000","last_clean":"2023-07-14T19:02:35.609345+0000","last_became_active":"2023-07-14T19:02:30.569461+0000","last_became_peered":"2023-07-14T19:02:30.569461+0000","last_unstale":"2023-07-14T19:02:35.609345+0000","last_undegraded":"2023-07-14T19:02:35.609345+0000","last_fullsized":"2023-07-14T19:02:35.609345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:35:17.252918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608767+0000","last_change":"2023-07-14T19:02:34.585220+0000","last_active":"2023-07-14T19:02:35.608767+0000","last_peered":"2023-07-14T19:02:35.608767+0000","last_clean":"2023-07-14T19:02:35.608767+0000","last_became_active":"2023-07-14T19:02:34.584771+0000","last_became_peered":"2023-07-14T19:02:34.584771+0000","last_unstale":"2023-07-14T19:02:35.608767+0000","last_undegraded":"2023-07-14T19:02:35.608767+0000","last_fullsized":"2023-07-14T19:02:35.608767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:48.195878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"80'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032869+0000","last_change":"2023-07-14T19:02:29.938571+0000","last_active":"2023-07-14T19:02:36.032869+0000","last_peered":"2023-07-14T19:02:36.032869+0000","last_clean":"2023-07-14T19:02:36.032869+0000","last_became_active":"2023-07-14T19:02:29.938270+0000","last_became_peered":"2023-07-14T19:02:29.938270+0000","last_unstale":"2023-07-14T19:02:36.032869+0000","last_undegraded":"2023-07-14T19:02:36.032869+0000","last_fullsized":"2023-07-14T19:02:36.032869+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:24:05.121677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811982+0000","last_change":"2023-07-14T19:02:30.575894+0000","last_active":"2023-07-14T19:02:35.811982+0000","last_peered":"2023-07-14T19:02:35.811982+0000","last_clean":"2023-07-14T19:02:35.811982+0000","last_became_active":"2023-07-14T19:02:30.575607+0000","last_became_peered":"2023-07-14T19:02:30.575607+0000","last_unstale":"2023-07-14T19:02:35.811982+0000","last_undegraded":"2023-07-14T19:02:35.811982+0000","last_fullsized":"2023-07-14T19:02:35.811982+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:30.612104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033103+0000","last_change":"2023-07-14T19:02:34.590761+0000","last_active":"2023-07-14T19:02:36.033103+0000","last_peered":"2023-07-14T19:02:36.033103+0000","last_clean":"2023-07-14T19:02:36.033103+0000","last_became_active":"2023-07-14T19:02:34.590304+0000","last_became_peered":"2023-07-14T19:02:34.590304+0000","last_unstale":"2023-07-14T19:02:36.033103+0000","last_undegraded":"2023-07-14T19:02:36.033103+0000","last_fullsized":"2023-07-14T19:02:36.033103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:18:06.211982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"80'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.812082+0000","last_change":"2023-07-14T19:02:29.944136+0000","last_active":"2023-07-14T19:02:35.812082+0000","last_peered":"2023-07-14T19:02:35.812082+0000","last_clean":"2023-07-14T19:02:35.812082+0000","last_became_active":"2023-07-14T19:02:29.943751+0000","last_became_peered":"2023-07-14T19:02:29.943751+0000","last_unstale":"2023-07-14T19:02:35.812082+0000","last_undegraded":"2023-07-14T19:02:35.812082+0000","last_fullsized":"2023-07-14T19:02:35.812082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:34:19.730141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032337+0000","last_change":"2023-07-14T19:02:30.583156+0000","last_active":"2023-07-14T19:02:36.032337+0000","last_peered":"2023-07-14T19:02:36.032337+0000","last_clean":"2023-07-14T19:02:36.032337+0000","last_became_active":"2023-07-14T19:02:30.582794+0000","last_became_peered":"2023-07-14T19:02:30.582794+0000","last_unstale":"2023-07-14T19:02:36.032337+0000","last_undegraded":"2023-07-14T19:02:36.032337+0000","last_fullsized":"2023-07-14T19:02:36.032337+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:48:34.691122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033132+0000","last_change":"2023-07-14T19:02:34.590788+0000","last_active":"2023-07-14T19:02:36.033132+0000","last_peered":"2023-07-14T19:02:36.033132+0000","last_clean":"2023-07-14T19:02:36.033132+0000","last_became_active":"2023-07-14T19:02:34.590385+0000","last_became_peered":"2023-07-14T19:02:34.590385+0000","last_unstale":"2023-07-14T19:02:36.033132+0000","last_undegraded":"2023-07-14T19:02:36.033132+0000","last_fullsized":"2023-07-14T19:02:36.033132+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:19:22.697138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.208154+0000","last_change":"2023-07-14T19:02:29.938180+0000","last_active":"2023-07-14T19:02:50.208154+0000","last_peered":"2023-07-14T19:02:50.208154+0000","last_clean":"2023-07-14T19:02:50.208154+0000","last_became_active":"2023-07-14T19:02:29.937823+0000","last_became_peered":"2023-07-14T19:02:29.937823+0000","last_unstale":"2023-07-14T19:02:50.208154+0000","last_undegraded":"2023-07-14T19:02:50.208154+0000","last_fullsized":"2023-07-14T19:02:50.208154+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:08:22.578643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607182+0000","last_change":"2023-07-14T19:02:30.582746+0000","last_active":"2023-07-14T19:02:35.607182+0000","last_peered":"2023-07-14T19:02:35.607182+0000","last_clean":"2023-07-14T19:02:35.607182+0000","last_became_active":"2023-07-14T19:02:30.582426+0000","last_became_peered":"2023-07-14T19:02:30.582426+0000","last_unstale":"2023-07-14T19:02:35.607182+0000","last_undegraded":"2023-07-14T19:02:35.607182+0000","last_fullsized":"2023-07-14T19:02:35.607182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:40.875764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034368+0000","last_change":"2023-07-14T19:02:34.589966+0000","last_active":"2023-07-14T19:02:36.034368+0000","last_peered":"2023-07-14T19:02:36.034368+0000","last_clean":"2023-07-14T19:02:36.034368+0000","last_became_active":"2023-07-14T19:02:34.589544+0000","last_became_peered":"2023-07-14T19:02:34.589544+0000","last_unstale":"2023-07-14T19:02:36.034368+0000","last_undegraded":"2023-07-14T19:02:36.034368+0000","last_fullsized":"2023-07-14T19:02:36.034368+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:02:51.191966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032844+0000","last_change":"2023-07-14T19:02:29.935507+0000","last_active":"2023-07-14T19:02:36.032844+0000","last_peered":"2023-07-14T19:02:36.032844+0000","last_clean":"2023-07-14T19:02:36.032844+0000","last_became_active":"2023-07-14T19:02:29.935160+0000","last_became_peered":"2023-07-14T19:02:29.935160+0000","last_unstale":"2023-07-14T19:02:36.032844+0000","last_undegraded":"2023-07-14T19:02:36.032844+0000","last_fullsized":"2023-07-14T19:02:36.032844+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:34:50.545076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034296+0000","last_change":"2023-07-14T19:02:30.578077+0000","last_active":"2023-07-14T19:02:36.034296+0000","last_peered":"2023-07-14T19:02:36.034296+0000","last_clean":"2023-07-14T19:02:36.034296+0000","last_became_active":"2023-07-14T19:02:30.577764+0000","last_became_peered":"2023-07-14T19:02:30.577764+0000","last_unstale":"2023-07-14T19:02:36.034296+0000","last_undegraded":"2023-07-14T19:02:36.034296+0000","last_fullsized":"2023-07-14T19:02:36.034296+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:20.545387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609074+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:02:35.609074+0000","last_peered":"2023-07-14T19:02:35.609074+0000","last_clean":"2023-07-14T19:02:35.609074+0000","last_became_active":"2023-07-14T19:02:34.583408+0000","last_became_peered":"2023-07-14T19:02:34.583408+0000","last_unstale":"2023-07-14T19:02:35.609074+0000","last_undegraded":"2023-07-14T19:02:35.609074+0000","last_fullsized":"2023-07-14T19:02:35.609074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:56:32.640321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:51.618467+0000","last_change":"2023-07-14T19:02:29.943899+0000","last_active":"2023-07-14T19:02:51.618467+0000","last_peered":"2023-07-14T19:02:51.618467+0000","last_clean":"2023-07-14T19:02:51.618467+0000","last_became_active":"2023-07-14T19:02:29.943585+0000","last_became_peered":"2023-07-14T19:02:29.943585+0000","last_unstale":"2023-07-14T19:02:51.618467+0000","last_undegraded":"2023-07-14T19:02:51.618467+0000","last_fullsized":"2023-07-14T19:02:51.618467+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:29.437187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033528+0000","last_change":"2023-07-14T19:02:30.567147+0000","last_active":"2023-07-14T19:02:36.033528+0000","last_peered":"2023-07-14T19:02:36.033528+0000","last_clean":"2023-07-14T19:02:36.033528+0000","last_became_active":"2023-07-14T19:02:30.566792+0000","last_became_peered":"2023-07-14T19:02:30.566792+0000","last_unstale":"2023-07-14T19:02:36.033528+0000","last_undegraded":"2023-07-14T19:02:36.033528+0000","last_fullsized":"2023-07-14T19:02:36.033528+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:14:15.802329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609090+0000","last_change":"2023-07-14T19:02:34.579561+0000","last_active":"2023-07-14T19:02:35.609090+0000","last_peered":"2023-07-14T19:02:35.609090+0000","last_clean":"2023-07-14T19:02:35.609090+0000","last_became_active":"2023-07-14T19:02:34.579099+0000","last_became_peered":"2023-07-14T19:02:34.579099+0000","last_unstale":"2023-07-14T19:02:35.609090+0000","last_undegraded":"2023-07-14T19:02:35.609090+0000","last_fullsized":"2023-07-14T19:02:35.609090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:24:00.439199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"80'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034320+0000","last_change":"2023-07-14T19:02:29.936651+0000","last_active":"2023-07-14T19:02:36.034320+0000","last_peered":"2023-07-14T19:02:36.034320+0000","last_clean":"2023-07-14T19:02:36.034320+0000","last_became_active":"2023-07-14T19:02:29.936280+0000","last_became_peered":"2023-07-14T19:02:29.936280+0000","last_unstale":"2023-07-14T19:02:36.034320+0000","last_undegraded":"2023-07-14T19:02:36.034320+0000","last_fullsized":"2023-07-14T19:02:36.034320+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:28.735813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033492+0000","last_change":"2023-07-14T19:02:30.575925+0000","last_active":"2023-07-14T19:02:36.033492+0000","last_peered":"2023-07-14T19:02:36.033492+0000","last_clean":"2023-07-14T19:02:36.033492+0000","last_became_active":"2023-07-14T19:02:30.575583+0000","last_became_peered":"2023-07-14T19:02:30.575583+0000","last_unstale":"2023-07-14T19:02:36.033492+0000","last_undegraded":"2023-07-14T19:02:36.033492+0000","last_fullsized":"2023-07-14T19:02:36.033492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:09:04.236418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033394+0000","last_change":"2023-07-14T19:02:34.587094+0000","last_active":"2023-07-14T19:02:36.033394+0000","last_peered":"2023-07-14T19:02:36.033394+0000","last_clean":"2023-07-14T19:02:36.033394+0000","last_became_active":"2023-07-14T19:02:34.586650+0000","last_became_peered":"2023-07-14T19:02:34.586650+0000","last_unstale":"2023-07-14T19:02:36.033394+0000","last_undegraded":"2023-07-14T19:02:36.033394+0000","last_fullsized":"2023-07-14T19:02:36.033394+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:42:08.985468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608485+0000","last_change":"2023-07-14T19:02:29.934651+0000","last_active":"2023-07-14T19:02:35.608485+0000","last_peered":"2023-07-14T19:02:35.608485+0000","last_clean":"2023-07-14T19:02:35.608485+0000","last_became_active":"2023-07-14T19:02:29.934347+0000","last_became_peered":"2023-07-14T19:02:29.934347+0000","last_unstale":"2023-07-14T19:02:35.608485+0000","last_undegraded":"2023-07-14T19:02:35.608485+0000","last_fullsized":"2023-07-14T19:02:35.608485+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:34:23.476531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811237+0000","last_change":"2023-07-14T19:02:30.571680+0000","last_active":"2023-07-14T19:02:35.811237+0000","last_peered":"2023-07-14T19:02:35.811237+0000","last_clean":"2023-07-14T19:02:35.811237+0000","last_became_active":"2023-07-14T19:02:30.571367+0000","last_became_peered":"2023-07-14T19:02:30.571367+0000","last_unstale":"2023-07-14T19:02:35.811237+0000","last_undegraded":"2023-07-14T19:02:35.811237+0000","last_fullsized":"2023-07-14T19:02:35.811237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:43.958053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032544+0000","last_change":"2023-07-14T19:02:34.577642+0000","last_active":"2023-07-14T19:02:36.032544+0000","last_peered":"2023-07-14T19:02:36.032544+0000","last_clean":"2023-07-14T19:02:36.032544+0000","last_became_active":"2023-07-14T19:02:34.577232+0000","last_became_peered":"2023-07-14T19:02:34.577232+0000","last_unstale":"2023-07-14T19:02:36.032544+0000","last_undegraded":"2023-07-14T19:02:36.032544+0000","last_fullsized":"2023-07-14T19:02:36.032544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:41:14.778029+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033710+0000","last_change":"2023-07-14T19:02:29.940688+0000","last_active":"2023-07-14T19:02:36.033710+0000","last_peered":"2023-07-14T19:02:36.033710+0000","last_clean":"2023-07-14T19:02:36.033710+0000","last_became_active":"2023-07-14T19:02:29.940444+0000","last_became_peered":"2023-07-14T19:02:29.940444+0000","last_unstale":"2023-07-14T19:02:36.033710+0000","last_undegraded":"2023-07-14T19:02:36.033710+0000","last_fullsized":"2023-07-14T19:02:36.033710+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:39:02.644668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032080+0000","last_change":"2023-07-14T19:02:30.568127+0000","last_active":"2023-07-14T19:02:36.032080+0000","last_peered":"2023-07-14T19:02:36.032080+0000","last_clean":"2023-07-14T19:02:36.032080+0000","last_became_active":"2023-07-14T19:02:30.567811+0000","last_became_peered":"2023-07-14T19:02:30.567811+0000","last_unstale":"2023-07-14T19:02:36.032080+0000","last_undegraded":"2023-07-14T19:02:36.032080+0000","last_fullsized":"2023-07-14T19:02:36.032080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:07:11.513097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811843+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:02:35.811843+0000","last_peered":"2023-07-14T19:02:35.811843+0000","last_clean":"2023-07-14T19:02:35.811843+0000","last_became_active":"2023-07-14T19:02:34.583480+0000","last_became_peered":"2023-07-14T19:02:34.583480+0000","last_unstale":"2023-07-14T19:02:35.811843+0000","last_undegraded":"2023-07-14T19:02:35.811843+0000","last_fullsized":"2023-07-14T19:02:35.811843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:27.737212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033996+0000","last_change":"2023-07-14T19:02:29.938459+0000","last_active":"2023-07-14T19:02:36.033996+0000","last_peered":"2023-07-14T19:02:36.033996+0000","last_clean":"2023-07-14T19:02:36.033996+0000","last_became_active":"2023-07-14T19:02:29.938212+0000","last_became_peered":"2023-07-14T19:02:29.938212+0000","last_unstale":"2023-07-14T19:02:36.033996+0000","last_undegraded":"2023-07-14T19:02:36.033996+0000","last_fullsized":"2023-07-14T19:02:36.033996+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:52:39.556855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607453+0000","last_change":"2023-07-14T19:02:30.586345+0000","last_active":"2023-07-14T19:02:35.607453+0000","last_peered":"2023-07-14T19:02:35.607453+0000","last_clean":"2023-07-14T19:02:35.607453+0000","last_became_active":"2023-07-14T19:02:30.585949+0000","last_became_peered":"2023-07-14T19:02:30.585949+0000","last_unstale":"2023-07-14T19:02:35.607453+0000","last_undegraded":"2023-07-14T19:02:35.607453+0000","last_fullsized":"2023-07-14T19:02:35.607453+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:34:26.531581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.031989+0000","last_change":"2023-07-14T19:02:34.582922+0000","last_active":"2023-07-14T19:02:36.031989+0000","last_peered":"2023-07-14T19:02:36.031989+0000","last_clean":"2023-07-14T19:02:36.031989+0000","last_became_active":"2023-07-14T19:02:34.582146+0000","last_became_peered":"2023-07-14T19:02:34.582146+0000","last_unstale":"2023-07-14T19:02:36.031989+0000","last_undegraded":"2023-07-14T19:02:36.031989+0000","last_fullsized":"2023-07-14T19:02:36.031989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:10:17.978844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.197064+0000","last_change":"2023-07-14T19:02:29.938215+0000","last_active":"2023-07-14T19:02:49.197064+0000","last_peered":"2023-07-14T19:02:49.197064+0000","last_clean":"2023-07-14T19:02:49.197064+0000","last_became_active":"2023-07-14T19:02:29.937889+0000","last_became_peered":"2023-07-14T19:02:29.937889+0000","last_unstale":"2023-07-14T19:02:49.197064+0000","last_undegraded":"2023-07-14T19:02:49.197064+0000","last_fullsized":"2023-07-14T19:02:49.197064+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:59:17.122842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032213+0000","last_change":"2023-07-14T19:02:30.569622+0000","last_active":"2023-07-14T19:02:36.032213+0000","last_peered":"2023-07-14T19:02:36.032213+0000","last_clean":"2023-07-14T19:02:36.032213+0000","last_became_active":"2023-07-14T19:02:30.569201+0000","last_became_peered":"2023-07-14T19:02:30.569201+0000","last_unstale":"2023-07-14T19:02:36.032213+0000","last_undegraded":"2023-07-14T19:02:36.032213+0000","last_fullsized":"2023-07-14T19:02:36.032213+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:27:36.907930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033572+0000","last_change":"2023-07-14T19:02:34.580978+0000","last_active":"2023-07-14T19:02:36.033572+0000","last_peered":"2023-07-14T19:02:36.033572+0000","last_clean":"2023-07-14T19:02:36.033572+0000","last_became_active":"2023-07-14T19:02:34.580503+0000","last_became_peered":"2023-07-14T19:02:34.580503+0000","last_unstale":"2023-07-14T19:02:36.033572+0000","last_undegraded":"2023-07-14T19:02:36.033572+0000","last_fullsized":"2023-07-14T19:02:36.033572+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:03:24.374548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.196704+0000","last_change":"2023-07-14T19:02:29.942400+0000","last_active":"2023-07-14T19:02:49.196704+0000","last_peered":"2023-07-14T19:02:49.196704+0000","last_clean":"2023-07-14T19:02:49.196704+0000","last_became_active":"2023-07-14T19:02:29.942080+0000","last_became_peered":"2023-07-14T19:02:29.942080+0000","last_unstale":"2023-07-14T19:02:49.196704+0000","last_undegraded":"2023-07-14T19:02:49.196704+0000","last_fullsized":"2023-07-14T19:02:49.196704+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:07:08.969267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607297+0000","last_change":"2023-07-14T19:02:30.582869+0000","last_active":"2023-07-14T19:02:35.607297+0000","last_peered":"2023-07-14T19:02:35.607297+0000","last_clean":"2023-07-14T19:02:35.607297+0000","last_became_active":"2023-07-14T19:02:30.582558+0000","last_became_peered":"2023-07-14T19:02:30.582558+0000","last_unstale":"2023-07-14T19:02:35.607297+0000","last_undegraded":"2023-07-14T19:02:35.607297+0000","last_fullsized":"2023-07-14T19:02:35.607297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:56:31.547355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033051+0000","last_change":"2023-07-14T19:02:34.582610+0000","last_active":"2023-07-14T19:02:36.033051+0000","last_peered":"2023-07-14T19:02:36.033051+0000","last_clean":"2023-07-14T19:02:36.033051+0000","last_became_active":"2023-07-14T19:02:34.581961+0000","last_became_peered":"2023-07-14T19:02:34.581961+0000","last_unstale":"2023-07-14T19:02:36.033051+0000","last_undegraded":"2023-07-14T19:02:36.033051+0000","last_fullsized":"2023-07-14T19:02:36.033051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:28:06.554521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'10","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:46.595988+0000","last_change":"2023-07-14T19:02:29.939314+0000","last_active":"2023-07-14T19:02:46.595988+0000","last_peered":"2023-07-14T19:02:46.595988+0000","last_clean":"2023-07-14T19:02:46.595988+0000","last_became_active":"2023-07-14T19:02:29.939000+0000","last_became_peered":"2023-07-14T19:02:29.939000+0000","last_unstale":"2023-07-14T19:02:46.595988+0000","last_undegraded":"2023-07-14T19:02:46.595988+0000","last_fullsized":"2023-07-14T19:02:46.595988+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:44:31.404753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3794,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033492+0000","last_change":"2023-07-14T19:02:30.572052+0000","last_active":"2023-07-14T19:02:36.033492+0000","last_peered":"2023-07-14T19:02:36.033492+0000","last_clean":"2023-07-14T19:02:36.033492+0000","last_became_active":"2023-07-14T19:02:30.571739+0000","last_became_peered":"2023-07-14T19:02:30.571739+0000","last_unstale":"2023-07-14T19:02:36.033492+0000","last_undegraded":"2023-07-14T19:02:36.033492+0000","last_fullsized":"2023-07-14T19:02:36.033492+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:23.810314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034605+0000","last_change":"2023-07-14T19:02:34.576663+0000","last_active":"2023-07-14T19:02:36.034605+0000","last_peered":"2023-07-14T19:02:36.034605+0000","last_clean":"2023-07-14T19:02:36.034605+0000","last_became_active":"2023-07-14T19:02:34.576162+0000","last_became_peered":"2023-07-14T19:02:34.576162+0000","last_unstale":"2023-07-14T19:02:36.034605+0000","last_undegraded":"2023-07-14T19:02:36.034605+0000","last_fullsized":"2023-07-14T19:02:36.034605+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:56:21.126418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.196082+0000","last_change":"2023-07-14T19:02:29.938859+0000","last_active":"2023-07-14T19:02:49.196082+0000","last_peered":"2023-07-14T19:02:49.196082+0000","last_clean":"2023-07-14T19:02:49.196082+0000","last_became_active":"2023-07-14T19:02:29.938630+0000","last_became_peered":"2023-07-14T19:02:29.938630+0000","last_unstale":"2023-07-14T19:02:49.196082+0000","last_undegraded":"2023-07-14T19:02:49.196082+0000","last_fullsized":"2023-07-14T19:02:49.196082+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:12:50.141569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608493+0000","last_change":"2023-07-14T19:02:30.573798+0000","last_active":"2023-07-14T19:02:35.608493+0000","last_peered":"2023-07-14T19:02:35.608493+0000","last_clean":"2023-07-14T19:02:35.608493+0000","last_became_active":"2023-07-14T19:02:30.572568+0000","last_became_peered":"2023-07-14T19:02:30.572568+0000","last_unstale":"2023-07-14T19:02:35.608493+0000","last_undegraded":"2023-07-14T19:02:35.608493+0000","last_fullsized":"2023-07-14T19:02:35.608493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:02.901707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032450+0000","last_change":"2023-07-14T19:02:34.582256+0000","last_active":"2023-07-14T19:02:36.032450+0000","last_peered":"2023-07-14T19:02:36.032450+0000","last_clean":"2023-07-14T19:02:36.032450+0000","last_became_active":"2023-07-14T19:02:34.581799+0000","last_became_peered":"2023-07-14T19:02:34.581799+0000","last_unstale":"2023-07-14T19:02:36.032450+0000","last_undegraded":"2023-07-14T19:02:36.032450+0000","last_fullsized":"2023-07-14T19:02:36.032450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:15:22.336363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.197291+0000","last_change":"2023-07-14T19:02:29.943874+0000","last_active":"2023-07-14T19:02:49.197291+0000","last_peered":"2023-07-14T19:02:49.197291+0000","last_clean":"2023-07-14T19:02:49.197291+0000","last_became_active":"2023-07-14T19:02:29.943596+0000","last_became_peered":"2023-07-14T19:02:29.943596+0000","last_unstale":"2023-07-14T19:02:49.197291+0000","last_undegraded":"2023-07-14T19:02:49.197291+0000","last_fullsized":"2023-07-14T19:02:49.197291+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:53:58.657606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032901+0000","last_change":"2023-07-14T19:02:30.572157+0000","last_active":"2023-07-14T19:02:36.032901+0000","last_peered":"2023-07-14T19:02:36.032901+0000","last_clean":"2023-07-14T19:02:36.032901+0000","last_became_active":"2023-07-14T19:02:30.571877+0000","last_became_peered":"2023-07-14T19:02:30.571877+0000","last_unstale":"2023-07-14T19:02:36.032901+0000","last_undegraded":"2023-07-14T19:02:36.032901+0000","last_fullsized":"2023-07-14T19:02:36.032901+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:16:17.711397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.031996+0000","last_change":"2023-07-14T19:02:34.588594+0000","last_active":"2023-07-14T19:02:36.031996+0000","last_peered":"2023-07-14T19:02:36.031996+0000","last_clean":"2023-07-14T19:02:36.031996+0000","last_became_active":"2023-07-14T19:02:34.587763+0000","last_became_peered":"2023-07-14T19:02:34.587763+0000","last_unstale":"2023-07-14T19:02:36.031996+0000","last_undegraded":"2023-07-14T19:02:36.031996+0000","last_fullsized":"2023-07-14T19:02:36.031996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:45.839894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:51.619051+0000","last_change":"2023-07-14T19:02:29.939145+0000","last_active":"2023-07-14T19:02:51.619051+0000","last_peered":"2023-07-14T19:02:51.619051+0000","last_clean":"2023-07-14T19:02:51.619051+0000","last_became_active":"2023-07-14T19:02:29.938816+0000","last_became_peered":"2023-07-14T19:02:29.938816+0000","last_unstale":"2023-07-14T19:02:51.619051+0000","last_undegraded":"2023-07-14T19:02:51.619051+0000","last_fullsized":"2023-07-14T19:02:51.619051+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:27:14.358986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032944+0000","last_change":"2023-07-14T19:02:30.572694+0000","last_active":"2023-07-14T19:02:36.032944+0000","last_peered":"2023-07-14T19:02:36.032944+0000","last_clean":"2023-07-14T19:02:36.032944+0000","last_became_active":"2023-07-14T19:02:30.572391+0000","last_became_peered":"2023-07-14T19:02:30.572391+0000","last_unstale":"2023-07-14T19:02:36.032944+0000","last_undegraded":"2023-07-14T19:02:36.032944+0000","last_fullsized":"2023-07-14T19:02:36.032944+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:19:15.638964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609130+0000","last_change":"2023-07-14T19:02:34.582476+0000","last_active":"2023-07-14T19:02:35.609130+0000","last_peered":"2023-07-14T19:02:35.609130+0000","last_clean":"2023-07-14T19:02:35.609130+0000","last_became_active":"2023-07-14T19:02:34.582083+0000","last_became_peered":"2023-07-14T19:02:34.582083+0000","last_unstale":"2023-07-14T19:02:35.609130+0000","last_undegraded":"2023-07-14T19:02:35.609130+0000","last_fullsized":"2023-07-14T19:02:35.609130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:53.732894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.210544+0000","last_change":"2023-07-14T19:02:29.941427+0000","last_active":"2023-07-14T19:02:50.210544+0000","last_peered":"2023-07-14T19:02:50.210544+0000","last_clean":"2023-07-14T19:02:50.210544+0000","last_became_active":"2023-07-14T19:02:29.941103+0000","last_became_peered":"2023-07-14T19:02:29.941103+0000","last_unstale":"2023-07-14T19:02:50.210544+0000","last_undegraded":"2023-07-14T19:02:50.210544+0000","last_fullsized":"2023-07-14T19:02:50.210544+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:57:09.780041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034128+0000","last_change":"2023-07-14T19:02:30.571778+0000","last_active":"2023-07-14T19:02:36.034128+0000","last_peered":"2023-07-14T19:02:36.034128+0000","last_clean":"2023-07-14T19:02:36.034128+0000","last_became_active":"2023-07-14T19:02:30.571478+0000","last_became_peered":"2023-07-14T19:02:30.571478+0000","last_unstale":"2023-07-14T19:02:36.034128+0000","last_undegraded":"2023-07-14T19:02:36.034128+0000","last_fullsized":"2023-07-14T19:02:36.034128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:41:32.785043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034066+0000","last_change":"2023-07-14T19:02:34.584170+0000","last_active":"2023-07-14T19:02:36.034066+0000","last_peered":"2023-07-14T19:02:36.034066+0000","last_clean":"2023-07-14T19:02:36.034066+0000","last_became_active":"2023-07-14T19:02:34.583773+0000","last_became_peered":"2023-07-14T19:02:34.583773+0000","last_unstale":"2023-07-14T19:02:36.034066+0000","last_undegraded":"2023-07-14T19:02:36.034066+0000","last_fullsized":"2023-07-14T19:02:36.034066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:52:47.607092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"80'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033133+0000","last_change":"2023-07-14T19:02:29.938386+0000","last_active":"2023-07-14T19:02:36.033133+0000","last_peered":"2023-07-14T19:02:36.033133+0000","last_clean":"2023-07-14T19:02:36.033133+0000","last_became_active":"2023-07-14T19:02:29.937394+0000","last_became_peered":"2023-07-14T19:02:29.937394+0000","last_unstale":"2023-07-14T19:02:36.033133+0000","last_undegraded":"2023-07-14T19:02:36.033133+0000","last_fullsized":"2023-07-14T19:02:36.033133+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:32.087374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033189+0000","last_change":"2023-07-14T19:02:30.572485+0000","last_active":"2023-07-14T19:02:36.033189+0000","last_peered":"2023-07-14T19:02:36.033189+0000","last_clean":"2023-07-14T19:02:36.033189+0000","last_became_active":"2023-07-14T19:02:30.572177+0000","last_became_peered":"2023-07-14T19:02:30.572177+0000","last_unstale":"2023-07-14T19:02:36.033189+0000","last_undegraded":"2023-07-14T19:02:36.033189+0000","last_fullsized":"2023-07-14T19:02:36.033189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:48:11.076249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032840+0000","last_change":"2023-07-14T19:02:34.582340+0000","last_active":"2023-07-14T19:02:36.032840+0000","last_peered":"2023-07-14T19:02:36.032840+0000","last_clean":"2023-07-14T19:02:36.032840+0000","last_became_active":"2023-07-14T19:02:34.581936+0000","last_became_peered":"2023-07-14T19:02:34.581936+0000","last_unstale":"2023-07-14T19:02:36.032840+0000","last_undegraded":"2023-07-14T19:02:36.032840+0000","last_fullsized":"2023-07-14T19:02:36.032840+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:01:12.341537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032203+0000","last_change":"2023-07-14T19:02:29.941404+0000","last_active":"2023-07-14T19:02:36.032203+0000","last_peered":"2023-07-14T19:02:36.032203+0000","last_clean":"2023-07-14T19:02:36.032203+0000","last_became_active":"2023-07-14T19:02:29.941052+0000","last_became_peered":"2023-07-14T19:02:29.941052+0000","last_unstale":"2023-07-14T19:02:36.032203+0000","last_undegraded":"2023-07-14T19:02:36.032203+0000","last_fullsized":"2023-07-14T19:02:36.032203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:50.738003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033689+0000","last_change":"2023-07-14T19:02:30.565671+0000","last_active":"2023-07-14T19:02:36.033689+0000","last_peered":"2023-07-14T19:02:36.033689+0000","last_clean":"2023-07-14T19:02:36.033689+0000","last_became_active":"2023-07-14T19:02:30.565375+0000","last_became_peered":"2023-07-14T19:02:30.565375+0000","last_unstale":"2023-07-14T19:02:36.033689+0000","last_undegraded":"2023-07-14T19:02:36.033689+0000","last_fullsized":"2023-07-14T19:02:36.033689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:00.244922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608815+0000","last_change":"2023-07-14T19:02:34.577692+0000","last_active":"2023-07-14T19:02:35.608815+0000","last_peered":"2023-07-14T19:02:35.608815+0000","last_clean":"2023-07-14T19:02:35.608815+0000","last_became_active":"2023-07-14T19:02:34.577329+0000","last_became_peered":"2023-07-14T19:02:34.577329+0000","last_unstale":"2023-07-14T19:02:35.608815+0000","last_undegraded":"2023-07-14T19:02:35.608815+0000","last_fullsized":"2023-07-14T19:02:35.608815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:42.179550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033959+0000","last_change":"2023-07-14T19:02:29.938330+0000","last_active":"2023-07-14T19:02:36.033959+0000","last_peered":"2023-07-14T19:02:36.033959+0000","last_clean":"2023-07-14T19:02:36.033959+0000","last_became_active":"2023-07-14T19:02:29.938022+0000","last_became_peered":"2023-07-14T19:02:29.938022+0000","last_unstale":"2023-07-14T19:02:36.033959+0000","last_undegraded":"2023-07-14T19:02:36.033959+0000","last_fullsized":"2023-07-14T19:02:36.033959+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:28:58.341498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033320+0000","last_change":"2023-07-14T19:02:30.569353+0000","last_active":"2023-07-14T19:02:36.033320+0000","last_peered":"2023-07-14T19:02:36.033320+0000","last_clean":"2023-07-14T19:02:36.033320+0000","last_became_active":"2023-07-14T19:02:30.569037+0000","last_became_peered":"2023-07-14T19:02:30.569037+0000","last_unstale":"2023-07-14T19:02:36.033320+0000","last_undegraded":"2023-07-14T19:02:36.033320+0000","last_fullsized":"2023-07-14T19:02:36.033320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:43.690826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609120+0000","last_change":"2023-07-14T19:02:34.580786+0000","last_active":"2023-07-14T19:02:35.609120+0000","last_peered":"2023-07-14T19:02:35.609120+0000","last_clean":"2023-07-14T19:02:35.609120+0000","last_became_active":"2023-07-14T19:02:34.580395+0000","last_became_peered":"2023-07-14T19:02:34.580395+0000","last_unstale":"2023-07-14T19:02:35.609120+0000","last_undegraded":"2023-07-14T19:02:35.609120+0000","last_fullsized":"2023-07-14T19:02:35.609120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:24.810182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034087+0000","last_change":"2023-07-14T19:02:30.574302+0000","last_active":"2023-07-14T19:02:36.034087+0000","last_peered":"2023-07-14T19:02:36.034087+0000","last_clean":"2023-07-14T19:02:36.034087+0000","last_became_active":"2023-07-14T19:02:30.573912+0000","last_became_peered":"2023-07-14T19:02:30.573912+0000","last_unstale":"2023-07-14T19:02:36.034087+0000","last_undegraded":"2023-07-14T19:02:36.034087+0000","last_fullsized":"2023-07-14T19:02:36.034087+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:12:45.492316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608417+0000","last_change":"2023-07-14T19:02:29.942302+0000","last_active":"2023-07-14T19:02:35.608417+0000","last_peered":"2023-07-14T19:02:35.608417+0000","last_clean":"2023-07-14T19:02:35.608417+0000","last_became_active":"2023-07-14T19:02:29.942005+0000","last_became_peered":"2023-07-14T19:02:29.942005+0000","last_unstale":"2023-07-14T19:02:35.608417+0000","last_undegraded":"2023-07-14T19:02:35.608417+0000","last_fullsized":"2023-07-14T19:02:35.608417+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:55.061870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033583+0000","last_change":"2023-07-14T19:02:34.578248+0000","last_active":"2023-07-14T19:02:36.033583+0000","last_peered":"2023-07-14T19:02:36.033583+0000","last_clean":"2023-07-14T19:02:36.033583+0000","last_became_active":"2023-07-14T19:02:34.577930+0000","last_became_peered":"2023-07-14T19:02:34.577930+0000","last_unstale":"2023-07-14T19:02:36.033583+0000","last_undegraded":"2023-07-14T19:02:36.033583+0000","last_fullsized":"2023-07-14T19:02:36.033583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:51:49.813630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811402+0000","last_change":"2023-07-14T19:02:30.582892+0000","last_active":"2023-07-14T19:02:35.811402+0000","last_peered":"2023-07-14T19:02:35.811402+0000","last_clean":"2023-07-14T19:02:35.811402+0000","last_became_active":"2023-07-14T19:02:30.582588+0000","last_became_peered":"2023-07-14T19:02:30.582588+0000","last_unstale":"2023-07-14T19:02:35.811402+0000","last_undegraded":"2023-07-14T19:02:35.811402+0000","last_fullsized":"2023-07-14T19:02:35.811402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:00:14.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.613182+0000","last_change":"2023-07-14T19:02:29.935857+0000","last_active":"2023-07-14T19:02:50.613182+0000","last_peered":"2023-07-14T19:02:50.613182+0000","last_clean":"2023-07-14T19:02:50.613182+0000","last_became_active":"2023-07-14T19:02:29.935489+0000","last_became_peered":"2023-07-14T19:02:29.935489+0000","last_unstale":"2023-07-14T19:02:50.613182+0000","last_undegraded":"2023-07-14T19:02:50.613182+0000","last_fullsized":"2023-07-14T19:02:50.613182+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:17:38.403976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":336,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032647+0000","last_change":"2023-07-14T19:01:31.967476+0000","last_active":"2023-07-14T19:02:36.032647+0000","last_peered":"2023-07-14T19:02:36.032647+0000","last_clean":"2023-07-14T19:02:36.032647+0000","last_became_active":"2023-07-14T19:01:31.967146+0000","last_became_peered":"2023-07-14T19:01:31.967146+0000","last_unstale":"2023-07-14T19:02:36.032647+0000","last_undegraded":"2023-07-14T19:02:36.032647+0000","last_fullsized":"2023-07-14T19:02:36.032647+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_clean_scrub_stamp":"2023-07-14T18:58:40.634620+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:05.217720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034257+0000","last_change":"2023-07-14T19:02:34.571823+0000","last_active":"2023-07-14T19:02:36.034257+0000","last_peered":"2023-07-14T19:02:36.034257+0000","last_clean":"2023-07-14T19:02:36.034257+0000","last_became_active":"2023-07-14T19:02:34.571359+0000","last_became_peered":"2023-07-14T19:02:34.571359+0000","last_unstale":"2023-07-14T19:02:36.034257+0000","last_undegraded":"2023-07-14T19:02:36.034257+0000","last_fullsized":"2023-07-14T19:02:36.034257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:33:12.956631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608304+0000","last_change":"2023-07-14T19:02:30.573736+0000","last_active":"2023-07-14T19:02:35.608304+0000","last_peered":"2023-07-14T19:02:35.608304+0000","last_clean":"2023-07-14T19:02:35.608304+0000","last_became_active":"2023-07-14T19:02:30.573381+0000","last_became_peered":"2023-07-14T19:02:30.573381+0000","last_unstale":"2023-07-14T19:02:35.608304+0000","last_undegraded":"2023-07-14T19:02:35.608304+0000","last_fullsized":"2023-07-14T19:02:35.608304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:21:36.599123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.209654+0000","last_change":"2023-07-14T19:02:29.937660+0000","last_active":"2023-07-14T19:02:50.209654+0000","last_peered":"2023-07-14T19:02:50.209654+0000","last_clean":"2023-07-14T19:02:50.209654+0000","last_became_active":"2023-07-14T19:02:29.937348+0000","last_became_peered":"2023-07-14T19:02:29.937348+0000","last_unstale":"2023-07-14T19:02:50.209654+0000","last_undegraded":"2023-07-14T19:02:50.209654+0000","last_fullsized":"2023-07-14T19:02:50.209654+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:35:12.947905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034302+0000","last_change":"2023-07-14T19:02:34.581534+0000","last_active":"2023-07-14T19:02:36.034302+0000","last_peered":"2023-07-14T19:02:36.034302+0000","last_clean":"2023-07-14T19:02:36.034302+0000","last_became_active":"2023-07-14T19:02:34.580998+0000","last_became_peered":"2023-07-14T19:02:34.580998+0000","last_unstale":"2023-07-14T19:02:36.034302+0000","last_undegraded":"2023-07-14T19:02:36.034302+0000","last_fullsized":"2023-07-14T19:02:36.034302+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:01:16.545480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.196866+0000","last_change":"2023-07-14T19:02:29.942336+0000","last_active":"2023-07-14T19:02:49.196866+0000","last_peered":"2023-07-14T19:02:49.196866+0000","last_clean":"2023-07-14T19:02:49.196866+0000","last_became_active":"2023-07-14T19:02:29.942033+0000","last_became_peered":"2023-07-14T19:02:29.942033+0000","last_unstale":"2023-07-14T19:02:49.196866+0000","last_undegraded":"2023-07-14T19:02:49.196866+0000","last_fullsized":"2023-07-14T19:02:49.196866+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:34.783218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608984+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:02:35.608984+0000","last_peered":"2023-07-14T19:02:35.608984+0000","last_clean":"2023-07-14T19:02:35.608984+0000","last_became_active":"2023-07-14T19:02:30.571129+0000","last_became_peered":"2023-07-14T19:02:30.571129+0000","last_unstale":"2023-07-14T19:02:35.608984+0000","last_undegraded":"2023-07-14T19:02:35.608984+0000","last_fullsized":"2023-07-14T19:02:35.608984+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:17.668104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811239+0000","last_change":"2023-07-14T19:02:34.580740+0000","last_active":"2023-07-14T19:02:35.811239+0000","last_peered":"2023-07-14T19:02:35.811239+0000","last_clean":"2023-07-14T19:02:35.811239+0000","last_became_active":"2023-07-14T19:02:34.580344+0000","last_became_peered":"2023-07-14T19:02:34.580344+0000","last_unstale":"2023-07-14T19:02:35.811239+0000","last_undegraded":"2023-07-14T19:02:35.811239+0000","last_fullsized":"2023-07-14T19:02:35.811239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:26:55.627423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"84'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032795+0000","last_change":"2023-07-14T19:02:29.941349+0000","last_active":"2023-07-14T19:02:36.032795+0000","last_peered":"2023-07-14T19:02:36.032795+0000","last_clean":"2023-07-14T19:02:36.032795+0000","last_became_active":"2023-07-14T19:02:29.941039+0000","last_became_peered":"2023-07-14T19:02:29.941039+0000","last_unstale":"2023-07-14T19:02:36.032795+0000","last_undegraded":"2023-07-14T19:02:36.032795+0000","last_fullsized":"2023-07-14T19:02:36.032795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:47:27.500876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811807+0000","last_change":"2023-07-14T19:02:30.577546+0000","last_active":"2023-07-14T19:02:35.811807+0000","last_peered":"2023-07-14T19:02:35.811807+0000","last_clean":"2023-07-14T19:02:35.811807+0000","last_became_active":"2023-07-14T19:02:30.576833+0000","last_became_peered":"2023-07-14T19:02:30.576833+0000","last_unstale":"2023-07-14T19:02:35.811807+0000","last_undegraded":"2023-07-14T19:02:35.811807+0000","last_fullsized":"2023-07-14T19:02:35.811807+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:47.060524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608552+0000","last_change":"2023-07-14T19:02:34.580655+0000","last_active":"2023-07-14T19:02:35.608552+0000","last_peered":"2023-07-14T19:02:35.608552+0000","last_clean":"2023-07-14T19:02:35.608552+0000","last_became_active":"2023-07-14T19:02:34.580186+0000","last_became_peered":"2023-07-14T19:02:34.580186+0000","last_unstale":"2023-07-14T19:02:35.608552+0000","last_undegraded":"2023-07-14T19:02:35.608552+0000","last_fullsized":"2023-07-14T19:02:35.608552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:30.211525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.196728+0000","last_change":"2023-07-14T19:02:29.938343+0000","last_active":"2023-07-14T19:02:49.196728+0000","last_peered":"2023-07-14T19:02:49.196728+0000","last_clean":"2023-07-14T19:02:49.196728+0000","last_became_active":"2023-07-14T19:02:29.938053+0000","last_became_peered":"2023-07-14T19:02:29.938053+0000","last_unstale":"2023-07-14T19:02:49.196728+0000","last_undegraded":"2023-07-14T19:02:49.196728+0000","last_fullsized":"2023-07-14T19:02:49.196728+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:19:06.212318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032632+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:02:36.032632+0000","last_peered":"2023-07-14T19:02:36.032632+0000","last_clean":"2023-07-14T19:02:36.032632+0000","last_became_active":"2023-07-14T19:02:30.572436+0000","last_became_peered":"2023-07-14T19:02:30.572436+0000","last_unstale":"2023-07-14T19:02:36.032632+0000","last_undegraded":"2023-07-14T19:02:36.032632+0000","last_fullsized":"2023-07-14T19:02:36.032632+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:58:47.846002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032889+0000","last_change":"2023-07-14T19:02:34.582900+0000","last_active":"2023-07-14T19:02:36.032889+0000","last_peered":"2023-07-14T19:02:36.032889+0000","last_clean":"2023-07-14T19:02:36.032889+0000","last_became_active":"2023-07-14T19:02:34.582493+0000","last_became_peered":"2023-07-14T19:02:34.582493+0000","last_unstale":"2023-07-14T19:02:36.032889+0000","last_undegraded":"2023-07-14T19:02:36.032889+0000","last_fullsized":"2023-07-14T19:02:36.032889+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:53:03.101901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.606993+0000","last_change":"2023-07-14T19:02:29.941819+0000","last_active":"2023-07-14T19:02:35.606993+0000","last_peered":"2023-07-14T19:02:35.606993+0000","last_clean":"2023-07-14T19:02:35.606993+0000","last_became_active":"2023-07-14T19:02:29.941469+0000","last_became_peered":"2023-07-14T19:02:29.941469+0000","last_unstale":"2023-07-14T19:02:35.606993+0000","last_undegraded":"2023-07-14T19:02:35.606993+0000","last_fullsized":"2023-07-14T19:02:35.606993+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:27.032813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032246+0000","last_change":"2023-07-14T19:02:30.565125+0000","last_active":"2023-07-14T19:02:36.032246+0000","last_peered":"2023-07-14T19:02:36.032246+0000","last_clean":"2023-07-14T19:02:36.032246+0000","last_became_active":"2023-07-14T19:02:30.564820+0000","last_became_peered":"2023-07-14T19:02:30.564820+0000","last_unstale":"2023-07-14T19:02:36.032246+0000","last_undegraded":"2023-07-14T19:02:36.032246+0000","last_fullsized":"2023-07-14T19:02:36.032246+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:37:29.561726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032950+0000","last_change":"2023-07-14T19:02:34.588273+0000","last_active":"2023-07-14T19:02:36.032950+0000","last_peered":"2023-07-14T19:02:36.032950+0000","last_clean":"2023-07-14T19:02:36.032950+0000","last_became_active":"2023-07-14T19:02:34.587202+0000","last_became_peered":"2023-07-14T19:02:34.587202+0000","last_unstale":"2023-07-14T19:02:36.032950+0000","last_undegraded":"2023-07-14T19:02:36.032950+0000","last_fullsized":"2023-07-14T19:02:36.032950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:24.908235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.195784+0000","last_change":"2023-07-14T19:02:29.938027+0000","last_active":"2023-07-14T19:02:49.195784+0000","last_peered":"2023-07-14T19:02:49.195784+0000","last_clean":"2023-07-14T19:02:49.195784+0000","last_became_active":"2023-07-14T19:02:29.937690+0000","last_became_peered":"2023-07-14T19:02:29.937690+0000","last_unstale":"2023-07-14T19:02:49.195784+0000","last_undegraded":"2023-07-14T19:02:49.195784+0000","last_fullsized":"2023-07-14T19:02:49.195784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:03.527958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032503+0000","last_change":"2023-07-14T19:02:30.576369+0000","last_active":"2023-07-14T19:02:36.032503+0000","last_peered":"2023-07-14T19:02:36.032503+0000","last_clean":"2023-07-14T19:02:36.032503+0000","last_became_active":"2023-07-14T19:02:30.576113+0000","last_became_peered":"2023-07-14T19:02:30.576113+0000","last_unstale":"2023-07-14T19:02:36.032503+0000","last_undegraded":"2023-07-14T19:02:36.032503+0000","last_fullsized":"2023-07-14T19:02:36.032503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:11:30.786937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":14,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:37.023177+0000","last_change":"2023-07-14T19:02:34.583724+0000","last_active":"2023-07-14T19:02:37.023177+0000","last_peered":"2023-07-14T19:02:37.023177+0000","last_clean":"2023-07-14T19:02:37.023177+0000","last_became_active":"2023-07-14T19:02:34.583218+0000","last_became_peered":"2023-07-14T19:02:34.583218+0000","last_unstale":"2023-07-14T19:02:37.023177+0000","last_undegraded":"2023-07-14T19:02:37.023177+0000","last_fullsized":"2023-07-14T19:02:37.023177+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:49:53.961252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608795+0000","last_change":"2023-07-14T19:02:29.938908+0000","last_active":"2023-07-14T19:02:35.608795+0000","last_peered":"2023-07-14T19:02:35.608795+0000","last_clean":"2023-07-14T19:02:35.608795+0000","last_became_active":"2023-07-14T19:02:29.938650+0000","last_became_peered":"2023-07-14T19:02:29.938650+0000","last_unstale":"2023-07-14T19:02:35.608795+0000","last_undegraded":"2023-07-14T19:02:35.608795+0000","last_fullsized":"2023-07-14T19:02:35.608795+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:19.121855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811550+0000","last_change":"2023-07-14T19:02:30.571870+0000","last_active":"2023-07-14T19:02:35.811550+0000","last_peered":"2023-07-14T19:02:35.811550+0000","last_clean":"2023-07-14T19:02:35.811550+0000","last_became_active":"2023-07-14T19:02:30.571521+0000","last_became_peered":"2023-07-14T19:02:30.571521+0000","last_unstale":"2023-07-14T19:02:35.811550+0000","last_undegraded":"2023-07-14T19:02:35.811550+0000","last_fullsized":"2023-07-14T19:02:35.811550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:37.931915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034509+0000","last_change":"2023-07-14T19:02:34.589900+0000","last_active":"2023-07-14T19:02:36.034509+0000","last_peered":"2023-07-14T19:02:36.034509+0000","last_clean":"2023-07-14T19:02:36.034509+0000","last_became_active":"2023-07-14T19:02:34.589481+0000","last_became_peered":"2023-07-14T19:02:34.589481+0000","last_unstale":"2023-07-14T19:02:36.034509+0000","last_undegraded":"2023-07-14T19:02:36.034509+0000","last_fullsized":"2023-07-14T19:02:36.034509+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:31:58.819621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.197810+0000","last_change":"2023-07-14T19:02:29.940315+0000","last_active":"2023-07-14T19:02:49.197810+0000","last_peered":"2023-07-14T19:02:49.197810+0000","last_clean":"2023-07-14T19:02:49.197810+0000","last_became_active":"2023-07-14T19:02:29.939991+0000","last_became_peered":"2023-07-14T19:02:29.939991+0000","last_unstale":"2023-07-14T19:02:49.197810+0000","last_undegraded":"2023-07-14T19:02:49.197810+0000","last_fullsized":"2023-07-14T19:02:49.197810+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:46.112090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034041+0000","last_change":"2023-07-14T19:02:30.583959+0000","last_active":"2023-07-14T19:02:36.034041+0000","last_peered":"2023-07-14T19:02:36.034041+0000","last_clean":"2023-07-14T19:02:36.034041+0000","last_became_active":"2023-07-14T19:02:30.583657+0000","last_became_peered":"2023-07-14T19:02:30.583657+0000","last_unstale":"2023-07-14T19:02:36.034041+0000","last_undegraded":"2023-07-14T19:02:36.034041+0000","last_fullsized":"2023-07-14T19:02:36.034041+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:38.543164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.606896+0000","last_change":"2023-07-14T19:02:34.579328+0000","last_active":"2023-07-14T19:02:35.606896+0000","last_peered":"2023-07-14T19:02:35.606896+0000","last_clean":"2023-07-14T19:02:35.606896+0000","last_became_active":"2023-07-14T19:02:34.578904+0000","last_became_peered":"2023-07-14T19:02:34.578904+0000","last_unstale":"2023-07-14T19:02:35.606896+0000","last_undegraded":"2023-07-14T19:02:35.606896+0000","last_fullsized":"2023-07-14T19:02:35.606896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:26:48.233983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609263+0000","last_change":"2023-07-14T19:02:29.940972+0000","last_active":"2023-07-14T19:02:35.609263+0000","last_peered":"2023-07-14T19:02:35.609263+0000","last_clean":"2023-07-14T19:02:35.609263+0000","last_became_active":"2023-07-14T19:02:29.940670+0000","last_became_peered":"2023-07-14T19:02:29.940670+0000","last_unstale":"2023-07-14T19:02:35.609263+0000","last_undegraded":"2023-07-14T19:02:35.609263+0000","last_fullsized":"2023-07-14T19:02:35.609263+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:09.182828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033918+0000","last_change":"2023-07-14T19:02:30.583714+0000","last_active":"2023-07-14T19:02:36.033918+0000","last_peered":"2023-07-14T19:02:36.033918+0000","last_clean":"2023-07-14T19:02:36.033918+0000","last_became_active":"2023-07-14T19:02:30.583404+0000","last_became_peered":"2023-07-14T19:02:30.583404+0000","last_unstale":"2023-07-14T19:02:36.033918+0000","last_undegraded":"2023-07-14T19:02:36.033918+0000","last_fullsized":"2023-07-14T19:02:36.033918+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:31.678407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032108+0000","last_change":"2023-07-14T19:02:34.582985+0000","last_active":"2023-07-14T19:02:36.032108+0000","last_peered":"2023-07-14T19:02:36.032108+0000","last_clean":"2023-07-14T19:02:36.032108+0000","last_became_active":"2023-07-14T19:02:34.582606+0000","last_became_peered":"2023-07-14T19:02:34.582606+0000","last_unstale":"2023-07-14T19:02:36.032108+0000","last_undegraded":"2023-07-14T19:02:36.032108+0000","last_fullsized":"2023-07-14T19:02:36.032108+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:22:30.700576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034472+0000","last_change":"2023-07-14T19:02:29.944064+0000","last_active":"2023-07-14T19:02:36.034472+0000","last_peered":"2023-07-14T19:02:36.034472+0000","last_clean":"2023-07-14T19:02:36.034472+0000","last_became_active":"2023-07-14T19:02:29.943770+0000","last_became_peered":"2023-07-14T19:02:29.943770+0000","last_unstale":"2023-07-14T19:02:36.034472+0000","last_undegraded":"2023-07-14T19:02:36.034472+0000","last_fullsized":"2023-07-14T19:02:36.034472+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:08:09.335177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034216+0000","last_change":"2023-07-14T19:02:30.580668+0000","last_active":"2023-07-14T19:02:36.034216+0000","last_peered":"2023-07-14T19:02:36.034216+0000","last_clean":"2023-07-14T19:02:36.034216+0000","last_became_active":"2023-07-14T19:02:30.580297+0000","last_became_peered":"2023-07-14T19:02:30.580297+0000","last_unstale":"2023-07-14T19:02:36.034216+0000","last_undegraded":"2023-07-14T19:02:36.034216+0000","last_fullsized":"2023-07-14T19:02:36.034216+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:31:20.673679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033333+0000","last_change":"2023-07-14T19:02:34.578894+0000","last_active":"2023-07-14T19:02:36.033333+0000","last_peered":"2023-07-14T19:02:36.033333+0000","last_clean":"2023-07-14T19:02:36.033333+0000","last_became_active":"2023-07-14T19:02:34.578463+0000","last_became_peered":"2023-07-14T19:02:34.578463+0000","last_unstale":"2023-07-14T19:02:36.033333+0000","last_undegraded":"2023-07-14T19:02:36.033333+0000","last_fullsized":"2023-07-14T19:02:36.033333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:59.507751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609171+0000","last_change":"2023-07-14T19:02:29.938430+0000","last_active":"2023-07-14T19:02:35.609171+0000","last_peered":"2023-07-14T19:02:35.609171+0000","last_clean":"2023-07-14T19:02:35.609171+0000","last_became_active":"2023-07-14T19:02:29.938118+0000","last_became_peered":"2023-07-14T19:02:29.938118+0000","last_unstale":"2023-07-14T19:02:35.609171+0000","last_undegraded":"2023-07-14T19:02:35.609171+0000","last_fullsized":"2023-07-14T19:02:35.609171+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:57:08.670837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033869+0000","last_change":"2023-07-14T19:02:30.575124+0000","last_active":"2023-07-14T19:02:36.033869+0000","last_peered":"2023-07-14T19:02:36.033869+0000","last_clean":"2023-07-14T19:02:36.033869+0000","last_became_active":"2023-07-14T19:02:30.574805+0000","last_became_peered":"2023-07-14T19:02:30.574805+0000","last_unstale":"2023-07-14T19:02:36.033869+0000","last_undegraded":"2023-07-14T19:02:36.033869+0000","last_fullsized":"2023-07-14T19:02:36.033869+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:36:50.721965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032645+0000","last_change":"2023-07-14T19:02:30.573897+0000","last_active":"2023-07-14T19:02:36.032645+0000","last_peered":"2023-07-14T19:02:36.032645+0000","last_clean":"2023-07-14T19:02:36.032645+0000","last_became_active":"2023-07-14T19:02:30.573585+0000","last_became_peered":"2023-07-14T19:02:30.573585+0000","last_unstale":"2023-07-14T19:02:36.032645+0000","last_undegraded":"2023-07-14T19:02:36.032645+0000","last_fullsized":"2023-07-14T19:02:36.032645+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:02:19.499089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608862+0000","last_change":"2023-07-14T19:02:30.572390+0000","last_active":"2023-07-14T19:02:35.608862+0000","last_peered":"2023-07-14T19:02:35.608862+0000","last_clean":"2023-07-14T19:02:35.608862+0000","last_became_active":"2023-07-14T19:02:30.572093+0000","last_became_peered":"2023-07-14T19:02:30.572093+0000","last_unstale":"2023-07-14T19:02:35.608862+0000","last_undegraded":"2023-07-14T19:02:35.608862+0000","last_fullsized":"2023-07-14T19:02:35.608862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:55:53.445808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608706+0000","last_change":"2023-07-14T19:02:30.572548+0000","last_active":"2023-07-14T19:02:35.608706+0000","last_peered":"2023-07-14T19:02:35.608706+0000","last_clean":"2023-07-14T19:02:35.608706+0000","last_became_active":"2023-07-14T19:02:30.572260+0000","last_became_peered":"2023-07-14T19:02:30.572260+0000","last_unstale":"2023-07-14T19:02:35.608706+0000","last_undegraded":"2023-07-14T19:02:35.608706+0000","last_fullsized":"2023-07-14T19:02:35.608706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:10.013952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032336+0000","last_change":"2023-07-14T19:02:30.569856+0000","last_active":"2023-07-14T19:02:36.032336+0000","last_peered":"2023-07-14T19:02:36.032336+0000","last_clean":"2023-07-14T19:02:36.032336+0000","last_became_active":"2023-07-14T19:02:30.569546+0000","last_became_peered":"2023-07-14T19:02:30.569546+0000","last_unstale":"2023-07-14T19:02:36.032336+0000","last_undegraded":"2023-07-14T19:02:36.032336+0000","last_fullsized":"2023-07-14T19:02:36.032336+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:27.292170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608964+0000","last_change":"2023-07-14T19:02:30.570836+0000","last_active":"2023-07-14T19:02:35.608964+0000","last_peered":"2023-07-14T19:02:35.608964+0000","last_clean":"2023-07-14T19:02:35.608964+0000","last_became_active":"2023-07-14T19:02:30.569675+0000","last_became_peered":"2023-07-14T19:02:30.569675+0000","last_unstale":"2023-07-14T19:02:35.608964+0000","last_undegraded":"2023-07-14T19:02:35.608964+0000","last_fullsized":"2023-07-14T19:02:35.608964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:50.028159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033623+0000","last_change":"2023-07-14T19:02:30.566846+0000","last_active":"2023-07-14T19:02:36.033623+0000","last_peered":"2023-07-14T19:02:36.033623+0000","last_clean":"2023-07-14T19:02:36.033623+0000","last_became_active":"2023-07-14T19:02:30.566525+0000","last_became_peered":"2023-07-14T19:02:30.566525+0000","last_unstale":"2023-07-14T19:02:36.033623+0000","last_undegraded":"2023-07-14T19:02:36.033623+0000","last_fullsized":"2023-07-14T19:02:36.033623+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:35:39.931938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609207+0000","last_change":"2023-07-14T19:02:30.579680+0000","last_active":"2023-07-14T19:02:35.609207+0000","last_peered":"2023-07-14T19:02:35.609207+0000","last_clean":"2023-07-14T19:02:35.609207+0000","last_became_active":"2023-07-14T19:02:30.579379+0000","last_became_peered":"2023-07-14T19:02:30.579379+0000","last_unstale":"2023-07-14T19:02:35.609207+0000","last_undegraded":"2023-07-14T19:02:35.609207+0000","last_fullsized":"2023-07-14T19:02:35.609207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:37:18.570317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811962+0000","last_change":"2023-07-14T19:02:30.577722+0000","last_active":"2023-07-14T19:02:35.811962+0000","last_peered":"2023-07-14T19:02:35.811962+0000","last_clean":"2023-07-14T19:02:35.811962+0000","last_became_active":"2023-07-14T19:02:30.577247+0000","last_became_peered":"2023-07-14T19:02:30.577247+0000","last_unstale":"2023-07-14T19:02:35.811962+0000","last_undegraded":"2023-07-14T19:02:35.811962+0000","last_fullsized":"2023-07-14T19:02:35.811962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:29.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033802+0000","last_change":"2023-07-14T19:02:30.571052+0000","last_active":"2023-07-14T19:02:36.033802+0000","last_peered":"2023-07-14T19:02:36.033802+0000","last_clean":"2023-07-14T19:02:36.033802+0000","last_became_active":"2023-07-14T19:02:30.570722+0000","last_became_peered":"2023-07-14T19:02:30.570722+0000","last_unstale":"2023-07-14T19:02:36.033802+0000","last_undegraded":"2023-07-14T19:02:36.033802+0000","last_fullsized":"2023-07-14T19:02:36.033802+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:08:38.060681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.812013+0000","last_change":"2023-07-14T19:02:30.575685+0000","last_active":"2023-07-14T19:02:35.812013+0000","last_peered":"2023-07-14T19:02:35.812013+0000","last_clean":"2023-07-14T19:02:35.812013+0000","last_became_active":"2023-07-14T19:02:30.575379+0000","last_became_peered":"2023-07-14T19:02:30.575379+0000","last_unstale":"2023-07-14T19:02:35.812013+0000","last_undegraded":"2023-07-14T19:02:35.812013+0000","last_fullsized":"2023-07-14T19:02:35.812013+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:31.763621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607133+0000","last_change":"2023-07-14T19:02:30.582450+0000","last_active":"2023-07-14T19:02:35.607133+0000","last_peered":"2023-07-14T19:02:35.607133+0000","last_clean":"2023-07-14T19:02:35.607133+0000","last_became_active":"2023-07-14T19:02:30.582074+0000","last_became_peered":"2023-07-14T19:02:30.582074+0000","last_unstale":"2023-07-14T19:02:35.607133+0000","last_undegraded":"2023-07-14T19:02:35.607133+0000","last_fullsized":"2023-07-14T19:02:35.607133+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:46.572335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608673+0000","last_change":"2023-07-14T19:02:30.573042+0000","last_active":"2023-07-14T19:02:35.608673+0000","last_peered":"2023-07-14T19:02:35.608673+0000","last_clean":"2023-07-14T19:02:35.608673+0000","last_became_active":"2023-07-14T19:02:30.572498+0000","last_became_peered":"2023-07-14T19:02:30.572498+0000","last_unstale":"2023-07-14T19:02:35.608673+0000","last_undegraded":"2023-07-14T19:02:35.608673+0000","last_fullsized":"2023-07-14T19:02:35.608673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:42:33.413533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033067+0000","last_change":"2023-07-14T19:02:30.569441+0000","last_active":"2023-07-14T19:02:36.033067+0000","last_peered":"2023-07-14T19:02:36.033067+0000","last_clean":"2023-07-14T19:02:36.033067+0000","last_became_active":"2023-07-14T19:02:30.569180+0000","last_became_peered":"2023-07-14T19:02:30.569180+0000","last_unstale":"2023-07-14T19:02:36.033067+0000","last_undegraded":"2023-07-14T19:02:36.033067+0000","last_fullsized":"2023-07-14T19:02:36.033067+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:26:58.302404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":7326,"num_objects":44,"num_object_clones":0,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":0,"num_read":21,"num_read_kb":17,"num_write":93,"num_write_kb":42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":188416,"data_stored":20258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514511,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":452924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.497}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84199999999999997}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]},{"osd":9,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74399999999999999}]},{"osd":10,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85799999999999998}]}]},{"osd":10,"up_from":69,"seq":296352743442,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28528,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716624,"statfs":{"total":95995035648,"available":95965822976,"internally_reserved":0,"allocated":1548288,"data_stored":1047280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.35099999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.621}]},{"osd":3,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.47799999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.72499999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.68999999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.65600000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75700000000000001}]}]},{"osd":9,"up_from":64,"seq":274877906966,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":933888,"data_stored":453506,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.439,"15min":0.439},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39000000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.63400000000000001},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.59499999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.35999999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.51600000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.58399999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.62},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.61599999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.752,"5min":0.752,"15min":0.752},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.629}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.49299999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]}]},{"osd":8,"up_from":57,"seq":244813135899,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":453331,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88300000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.60299999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90900000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76300000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.72899999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.63500000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.878,"15min":0.878},"last":0.78300000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.88500000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.68400000000000005,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.92700000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.97899999999999998},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.626,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.997}]},{"osd":6,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.94299999999999995,"5min":1.01,"15min":1.01},"last":0.90900000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.871,"15min":0.871},"last":0.84999999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0369999999999999,"5min":1.046,"15min":1.046},"last":0.93999999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:02:11 2023","interfaces":[{"interface":"back","average":{"1min":0.748,"5min":0.748,"15min":0.748},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.0189999999999999,"5min":1.0189999999999999,"15min":1.0189999999999999},"last":1.036},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":0.96699999999999997,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.877}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":1.016},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":1.0700000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]}]},{"osd":7,"up_from":50,"seq":214748364832,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28516,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716636,"statfs":{"total":95995035648,"available":95965835264,"internally_reserved":0,"allocated":1536000,"data_stored":1044715,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.39500000000000002,"15min":0.38900000000000001},"min":{"1min":0.28000000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63600000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.41099999999999998,"15min":0.40400000000000003},"min":{"1min":0.30299999999999999,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41599999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.55500000000000005,"15min":0.54300000000000004},"min":{"1min":0.41499999999999998,"5min":0.31,"15min":0.31},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.54500000000000004,"15min":0.53900000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.75800000000000001,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.53900000000000003,"15min":0.52900000000000003},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76700000000000002,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.622}]},{"osd":4,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46000000000000002,"15min":0.45300000000000001},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.49099999999999999,"15min":0.48499999999999999},"min":{"1min":0.33800000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.44500000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46800000000000003,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.77400000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61099999999999999,"15min":0.60099999999999998},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.64300000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.63400000000000001,"15min":0.627},"min":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.72799999999999998,"5min":0.65000000000000002,"15min":0.63700000000000001},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72399999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.66300000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:02:07 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"max":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.69399999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"min":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.74099999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528356,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":452924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.40100000000000002,"15min":0.38700000000000001},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":1.1439999999999999,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.497,"5min":0.42999999999999999,"15min":0.41899999999999998},"min":{"1min":0.28399999999999997,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":1.591,"5min":1.591,"15min":1.591},"last":0.30099999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.72999999999999998,"5min":0.57899999999999996,"15min":0.55400000000000005},"min":{"1min":0.40200000000000002,"5min":0.376,"15min":0.376},"max":{"1min":1.159,"5min":1.159,"15min":1.159},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.55300000000000005,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.49,"5min":1.49,"15min":1.49},"last":0.48599999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.60499999999999998,"5min":0.59299999999999997,"15min":0.59099999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.3680000000000001,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.60799999999999998,"15min":0.59199999999999997},"min":{"1min":0.46300000000000002,"5min":0.435,"15min":0.435},"max":{"1min":1.419,"5min":1.419,"15min":1.419},"last":0.70999999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.50600000000000001,"15min":0.49399999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.2689999999999999,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.5,"15min":0.49199999999999999},"min":{"1min":0.32300000000000001,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":2.077,"5min":2.077,"15min":2.077},"last":0.40300000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.59699999999999998,"15min":0.58199999999999996},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.76000000000000001,"5min":0.64300000000000002,"15min":0.623},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.1990000000000001,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.53700000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:02:04 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59799999999999998,"15min":0.58999999999999997},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.3879999999999999,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.65700000000000003,"15min":0.65500000000000003},"min":{"1min":0.48799999999999999,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":1.2230000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.51900000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:02:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68600000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.76200000000000001,"15min":0.73499999999999999},"min":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.6709999999999998,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.628}]},{"osd":8,"last update":"Fri Jul 14 19:01:46 2023","interfaces":[{"interface":"back","average":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"min":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":1.0549999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.62}]},{"osd":9,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.501},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.5309999999999999,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.44600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:02:31 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.66400000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67800000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574904,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":456718,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.5,"15min":0.48599999999999999},"min":{"1min":0.39300000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.441},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.50800000000000001,"15min":0.504},"min":{"1min":0.377,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.66200000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.503}]},{"osd":2,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.55200000000000005,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.52500000000000002,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.67300000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.57099999999999995,"15min":0.55300000000000005},"min":{"1min":0.44800000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.79800000000000004,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.56499999999999995,"15min":0.55200000000000005},"min":{"1min":0.433,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81299999999999994,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.77400000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:02:25 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.40500000000000003,"15min":0.377},"min":{"1min":0.30399999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.41399999999999998,"15min":0.39200000000000002},"min":{"1min":0.34000000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.47999999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.69999999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.59599999999999997,"15min":0.58499999999999996},"min":{"1min":0.46300000000000002,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59199999999999997,"15min":0.58299999999999996},"min":{"1min":0.40600000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.72499999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.52100000000000002,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.58299999999999996,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68600000000000005}]},{"osd":8,"last update":"Fri Jul 14 19:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"min":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.016,"5min":1.016,"15min":1.016},"last":0.91400000000000003},{"interface":"front","average":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.93600000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.054,"5min":1.054,"15min":1.054},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.83299999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.89500000000000002},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.63500000000000001}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":454129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.53100000000000003,"15min":0.503},"min":{"1min":0.371,"5min":0.159,"15min":0.159},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.626},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.502,"15min":0.48399999999999999},"min":{"1min":0.34999999999999998,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.496}]},{"osd":2,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.37,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.746,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.58899999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.435,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.29499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.33200000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:02:24 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.54200000000000004,"15min":0.52300000000000002},"min":{"1min":0.42299999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.52200000000000002,"15min":0.5},"min":{"1min":0.46400000000000002,"5min":0.309,"15min":0.309},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.60999999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.46000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91700000000000004,"5min":1.042,"15min":1.042},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.41799999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.504,"15min":0.498},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.66300000000000003,"5min":0.58599999999999997,"15min":0.57299999999999995},"min":{"1min":0.372,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.1359999999999999,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.47599999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.67000000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.74199999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70199999999999996,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.024,"15min":1.024},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.66400000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.204,"5min":1.204,"15min":1.204},"last":0.78200000000000003},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.55600000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"min":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"max":{"1min":1.2549999999999999,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.56699999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88}]}]},{"osd":2,"up_from":18,"seq":77309411381,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":954368,"data_stored":457326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78700000000000003,"5min":1.111,"15min":1.111},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.82399999999999995}]},{"osd":1,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.628,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.55600000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53900000000000003,"15min":0.53600000000000003},"min":{"1min":0.34699999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.77600000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.752},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.27800000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.628}]},{"osd":4,"last update":"Fri Jul 14 19:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.69999999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.41599999999999998,"15min":0.40799999999999997},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.378},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.46999999999999997,"15min":0.46200000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.433}]},{"osd":6,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.65900000000000003,"5min":0.57199999999999995,"15min":0.55800000000000005},"min":{"1min":0.41099999999999998,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.91900000000000004,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.88},{"interface":"front","average":{"1min":0.63,"5min":0.58999999999999997,"15min":0.58399999999999996},"min":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.96199999999999997,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.72199999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60199999999999998,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61599999999999999,"15min":0.60699999999999998},"min":{"1min":0.47499999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64800000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:01:48 2023","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.59999999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.68100000000000005,"15min":0.68100000000000005},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.0940000000000001,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"min":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.0820000000000001,"5min":1.0820000000000001,"15min":1.0820000000000001},"last":0.80600000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.76600000000000001},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.85599999999999998}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83999999999999997}]}]},{"osd":3,"up_from":25,"seq":107374182448,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":937984,"data_stored":453028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.33800000000000002,"15min":0.32900000000000001},"min":{"1min":0.26600000000000001,"5min":0.188,"15min":0.188},"max":{"1min":0.54800000000000004,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.371,"5min":0.34999999999999998,"15min":0.34200000000000003},"min":{"1min":0.23599999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.32400000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.315,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76900000000000002,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.41399999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.69999999999999996,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.626}]},{"osd":2,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.63900000000000001,"15min":0.66300000000000003},"min":{"1min":0.41499999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.82499999999999996,"5min":4.6509999999999998,"15min":4.6509999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.63600000000000001,"15min":0.67300000000000004},"min":{"1min":0.41599999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79100000000000004,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.56100000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:02:25 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.53400000000000003,"15min":0.52500000000000002},"min":{"1min":0.36599999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.39200000000000002,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.49099999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.498,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.64000000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.48799999999999999,"15min":0.48399999999999999},"min":{"1min":0.33700000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.54400000000000004,"15min":0.53800000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.86599999999999999,"5min":1.119,"15min":1.119},"last":0.73999999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76500000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.48999999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.53500000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:02:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.69099999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.67600000000000005}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66600000000000004}]}]},{"osd":4,"up_from":33,"seq":141733920813,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":453637,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.503,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.38100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.36299999999999999,"15min":0.34100000000000003},"min":{"1min":0.29999999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.90500000000000003,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.377,"15min":0.35699999999999998},"min":{"1min":0.255,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.751,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.45100000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.54700000000000004,"15min":0.53100000000000003},"min":{"1min":0.42599999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.95199999999999996,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.56100000000000005,"15min":0.53500000000000003},"min":{"1min":0.40200000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.105,"5min":1.105,"15min":1.105},"last":0.54300000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.53100000000000003,"15min":0.505},"min":{"1min":0.36099999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.93700000000000006,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.625,"5min":0.52600000000000002,"15min":0.498},"min":{"1min":0.38100000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.71799999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57599999999999996,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.59399999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.751},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.60799999999999998,"15min":0.59999999999999998},"min":{"1min":0.45000000000000001,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.625}]},{"osd":7,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.56399999999999995,"15min":0.55100000000000005},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.57099999999999995,"15min":0.56699999999999995},"min":{"1min":0.39400000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.80500000000000005,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.81599999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68600000000000005,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.78000000000000003},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.64900000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:02:08 2023","interfaces":[{"interface":"back","average":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.94999999999999996,"5min":0.94999999999999996,"15min":0.94999999999999996},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.73799999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.86199999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79500000000000004}]}]},{"osd":5,"up_from":38,"seq":163208757288,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28532,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716620,"statfs":{"total":95995035648,"available":95965818880,"internally_reserved":0,"allocated":1552384,"data_stored":1048875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.52400000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.59399999999999997}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.5},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.41399999999999998,"15min":0.38900000000000001},"min":{"1min":0.309,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.56200000000000006}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70599999999999996,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.67100000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.874},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.78000000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.65400000000000003,"5min":0.59599999999999997,"15min":0.58599999999999997},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.68500000000000005,"5min":0.63700000000000001,"15min":0.629},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.64300000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:02:26 2023","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61899999999999999,"15min":0.61499999999999999},"min":{"1min":0.47599999999999998,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.629,"5min":0.63300000000000001,"15min":0.63400000000000001},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.77900000000000003,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.89000000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.754},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.90400000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.753,"5min":0.753,"15min":0.753},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1,"5min":1,"15min":1},"last":0.81799999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.85799999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":3988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:02:53.619 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-07-14T19:02:53.620 INFO:tasks.cephadm:Setting up client nodes... 2023-07-14T19:02:53.621 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-07-14T19:02:53.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:53 smithi102 ceph-mon[129280]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:53.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:53 smithi102 ceph-mon[129280]: pgmap v266: 129 pgs: 129 active+clean; 584 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 2.4 KiB/s wr, 5 op/s 2023-07-14T19:02:53.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:53 smithi102 ceph-mon[129280]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:02:53.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:53 smithi102 ceph-mon[129280]: mgrmap e17: x(active, since 6m), standbys: z, y 2023-07-14T19:02:53.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:53 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3705707604' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:02:53.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:53 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/26879894' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:02:54.201 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:02:54.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:54 smithi088 ceph-mon[128973]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:02:54.464 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:54 smithi078 ceph-mon[127425]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:02:54.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:54 smithi102 ceph-mon[129280]: from='client.14919 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:02:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:55 smithi078 ceph-mon[127425]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:02:55.284 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:55 smithi078 ceph-mon[127425]: pgmap v267: 129 pgs: 129 active+clean; 618 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2023-07-14T19:02:55.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:55 smithi088 ceph-mon[128973]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:02:55.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:55 smithi088 ceph-mon[128973]: pgmap v267: 129 pgs: 129 active+clean; 618 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2023-07-14T19:02:55.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 -- 172.21.15.78:0/877434001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 msgr2=0x7fd34c0ff9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:55.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 --2- 172.21.15.78:0/877434001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 0x7fd34c0ff9d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd34800aaa0 tx=0x7fd34800adb0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 -- 172.21.15.78:0/877434001 shutdown_connections 2023-07-14T19:02:55.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 --2- 172.21.15.78:0/877434001 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd34c0fe380 0x7fd34c0fe790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 --2- 172.21.15.78:0/877434001 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd34c0fff10 0x7fd34c1067d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 --2- 172.21.15.78:0/877434001 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 0x7fd34c0ff9d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 -- 172.21.15.78:0/877434001 >> 172.21.15.78:0/877434001 conn(0x7fd34c0f99b0 msgr2=0x7fd34c0fbe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:55.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 -- 172.21.15.78:0/877434001 shutdown_connections 2023-07-14T19:02:55.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.516+0000 7fd3532da700 1 -- 172.21.15.78:0/877434001 wait complete. 2023-07-14T19:02:55.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd3532da700 1 Processor -- start 2023-07-14T19:02:55.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd3532da700 1 -- start start 2023-07-14T19:02:55.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd3532da700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd34c0fe380 0x7fd34c102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:55.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd3532da700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 0x7fd34c100960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:55.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd3532da700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd34c0fff10 0x7fd34c100ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:55.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd3532da700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c101620 con 0x7fd34c0ff580 2023-07-14T19:02:55.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd3532da700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c101790 con 0x7fd34c0fff10 2023-07-14T19:02:55.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd3532da700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd34c19f9b0 con 0x7fd34c0fe380 2023-07-14T19:02:55.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd350875700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 0x7fd34c100960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:55.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd350875700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 0x7fd34c100960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35058/0 (socket says 172.21.15.78:35058) 2023-07-14T19:02:55.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd350875700 1 -- 172.21.15.78:0/1115126976 learned_addr learned my addr 172.21.15.78:0/1115126976 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:55.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd351076700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd34c0fe380 0x7fd34c102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:55.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd350875700 1 -- 172.21.15.78:0/1115126976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd34c0fe380 msgr2=0x7fd34c102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd350875700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd34c0fe380 0x7fd34c102310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd350875700 1 -- 172.21.15.78:0/1115126976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd34c0fff10 msgr2=0x7fd34c100ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd350875700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd34c0fff10 0x7fd34c100ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.517+0000 7fd350875700 1 -- 172.21.15.78:0/1115126976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd34800a700 con 0x7fd34c0ff580 2023-07-14T19:02:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.518+0000 7fd350875700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 0x7fd34c100960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd3480054c0 tx=0x7fd3480055a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:55.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.518+0000 7fd3427fc700 1 -- 172.21.15.78:0/1115126976 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd34800f6e0 con 0x7fd34c0ff580 2023-07-14T19:02:55.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.518+0000 7fd3427fc700 1 -- 172.21.15.78:0/1115126976 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd34800fcd0 con 0x7fd34c0ff580 2023-07-14T19:02:55.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.518+0000 7fd3427fc700 1 -- 172.21.15.78:0/1115126976 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd348016bd0 con 0x7fd34c0ff580 2023-07-14T19:02:55.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.518+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd34c19fc30 con 0x7fd34c0ff580 2023-07-14T19:02:55.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.518+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd34c1a0100 con 0x7fd34c0ff580 2023-07-14T19:02:55.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.519+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34c109200 con 0x7fd34c0ff580 2023-07-14T19:02:55.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.520+0000 7fd3427fc700 1 -- 172.21.15.78:0/1115126976 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd348016d30 con 0x7fd34c0ff580 2023-07-14T19:02:55.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.520+0000 7fd3427fc700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd3380a11e0 0x7fd3380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:55.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.520+0000 7fd351076700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd3380a11e0 0x7fd3380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:55.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.521+0000 7fd3427fc700 1 -- 172.21.15.78:0/1115126976 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fd348012070 con 0x7fd34c0ff580 2023-07-14T19:02:55.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.521+0000 7fd351076700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd3380a11e0 0x7fd3380a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd33c006fd0 tx=0x7fd33c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:55.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.524+0000 7fd3427fc700 1 -- 172.21.15.78:0/1115126976 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd348098930 con 0x7fd34c0ff580 2023-07-14T19:02:55.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:55 smithi102 ceph-mon[129280]: from='client.14925 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:02:55.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:55 smithi102 ceph-mon[129280]: pgmap v267: 129 pgs: 129 active+clean; 618 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2023-07-14T19:02:55.789 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.785+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fd34c1a06b0 con 0x7fd34c0ff580 2023-07-14T19:02:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.791+0000 7fd3427fc700 1 -- 172.21.15.78:0/1115126976 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v31) v1 ==== 170+0+59 (secure 0 0 0) 0x7fd34809bf50 con 0x7fd34c0ff580 2023-07-14T19:02:55.795 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2023-07-14T19:02:55.796 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQDfm7FkpVbkLhAA7fY0hmk2ENtlmnPpi1xWcw== 2023-07-14T19:02:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd3380a11e0 msgr2=0x7fd3380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd3380a11e0 0x7fd3380a3690 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd33c006fd0 tx=0x7fd33c008040 comp rx=0 tx=0).stop 2023-07-14T19:02:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 msgr2=0x7fd34c100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 0x7fd34c100960 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd3480054c0 tx=0x7fd3480055a0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 shutdown_connections 2023-07-14T19:02:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd3380a11e0 0x7fd3380a3690 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd34c0fff10 0x7fd34c100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd34c0fe380 0x7fd34c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 --2- 172.21.15.78:0/1115126976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd34c0ff580 0x7fd34c100960 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:55.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 >> 172.21.15.78:0/1115126976 conn(0x7fd34c0f99b0 msgr2=0x7fd34c0fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:55.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 shutdown_connections 2023-07-14T19:02:55.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:55.794+0000 7fd3532da700 1 -- 172.21.15.78:0/1115126976 wait complete. 2023-07-14T19:02:56.263 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T19:02:56.264 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-07-14T19:02:56.264 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-07-14T19:02:56.305 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-07-14T19:02:56.305 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-07-14T19:02:56.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph mgr dump --format=json 2023-07-14T19:02:56.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:56 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1115126976' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:02:56.881 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:56 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1115126976' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:02:56.882 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:56 smithi102 ceph-mon[129280]: pgmap v268: 129 pgs: 129 active+clean; 619 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2023-07-14T19:02:56.890 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:02:56.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:56 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1115126976' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:02:56.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:56 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1115126976' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:02:56.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:56 smithi088 ceph-mon[128973]: pgmap v268: 129 pgs: 129 active+clean; 619 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2023-07-14T19:02:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:56 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1115126976' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-07-14T19:02:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:56 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1115126976' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-07-14T19:02:57.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:56 smithi078 ceph-mon[127425]: pgmap v268: 129 pgs: 129 active+clean; 619 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.6 KiB/s wr, 11 op/s 2023-07-14T19:02:58.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.099+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1089418711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc104ca0 msgr2=0x7f3bcc105070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:58.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.099+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1089418711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc104ca0 0x7f3bcc105070 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009990 tx=0x7f3bbc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.100+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1089418711 shutdown_connections 2023-07-14T19:02:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.100+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1089418711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc0febf0 0x7f3bcc10cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.100+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1089418711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc104ca0 0x7f3bcc105070 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.100+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1089418711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bcc0fe240 0x7f3bcc0fe6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.100+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1089418711 >> 172.21.15.78:0/1089418711 conn(0x7f3bcc075050 msgr2=0x7f3bcc075450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.100+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1089418711 shutdown_connections 2023-07-14T19:02:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.100+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1089418711 wait complete. 2023-07-14T19:02:58.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.100+0000 7f3bd12c8700 1 Processor -- start 2023-07-14T19:02:58.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bd12c8700 1 -- start start 2023-07-14T19:02:58.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bd12c8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bcc0fe240 0x7f3bcc19bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:58.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bd12c8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc0febf0 0x7f3bcc19c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:58.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bd12c8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc104ca0 0x7f3bcc195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bd12c8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc1961c0 con 0x7f3bcc0febf0 2023-07-14T19:02:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bd12c8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc196330 con 0x7f3bcc0fe240 2023-07-14T19:02:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bd12c8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bcc1964f0 con 0x7f3bcc104ca0 2023-07-14T19:02:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bca7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc0febf0 0x7f3bcc19c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bca7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc0febf0 0x7f3bcc19c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35070/0 (socket says 172.21.15.78:35070) 2023-07-14T19:02:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bca7fc700 1 -- 172.21.15.78:0/1185364340 learned_addr learned my addr 172.21.15.78:0/1185364340 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:02:58.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bcaffd700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bcc0fe240 0x7f3bcc19bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bcb7fe700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc104ca0 0x7f3bcc195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bca7fc700 1 -- 172.21.15.78:0/1185364340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc104ca0 msgr2=0x7f3bcc195b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bca7fc700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc104ca0 0x7f3bcc195b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bca7fc700 1 -- 172.21.15.78:0/1185364340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bcc0fe240 msgr2=0x7f3bcc19bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:58.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bca7fc700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bcc0fe240 0x7f3bcc19bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bca7fc700 1 -- 172.21.15.78:0/1185364340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bbc0095f0 con 0x7f3bcc0febf0 2023-07-14T19:02:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.101+0000 7f3bcb7fe700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc104ca0 0x7f3bcc195b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:02:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.102+0000 7f3bca7fc700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc0febf0 0x7f3bcc19c010 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000d950 tx=0x7f3bc000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.102+0000 7f3bb3fff700 1 -- 172.21.15.78:0/1185364340 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc0009940 con 0x7f3bcc0febf0 2023-07-14T19:02:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.102+0000 7f3bb3fff700 1 -- 172.21.15.78:0/1185364340 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3bc0010460 con 0x7f3bcc0febf0 2023-07-14T19:02:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.102+0000 7f3bb3fff700 1 -- 172.21.15.78:0/1185364340 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc000f620 con 0x7f3bcc0febf0 2023-07-14T19:02:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.102+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bcc1967d0 con 0x7f3bcc0febf0 2023-07-14T19:02:58.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.102+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bcc1a2ac0 con 0x7f3bcc0febf0 2023-07-14T19:02:58.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.103+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bcc04e340 con 0x7f3bcc0febf0 2023-07-14T19:02:58.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.104+0000 7f3bb3fff700 1 -- 172.21.15.78:0/1185364340 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3bc00105d0 con 0x7f3bcc0febf0 2023-07-14T19:02:58.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.104+0000 7f3bb3fff700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3bb40a11e0 0x7f3bb40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:02:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.104+0000 7f3bcaffd700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3bb40a11e0 0x7f3bb40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:02:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.104+0000 7f3bb3fff700 1 -- 172.21.15.78:0/1185364340 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3bc00ccae0 con 0x7f3bcc0febf0 2023-07-14T19:02:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.105+0000 7f3bcaffd700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3bb40a11e0 0x7f3bb40a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00b5c0 tx=0x7f3bbc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:02:58.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.108+0000 7f3bb3fff700 1 -- 172.21.15.78:0/1185364340 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3bc0095ee0 con 0x7f3bcc0febf0 2023-07-14T19:02:58.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.351+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f3bcc1a2c50 con 0x7f3bcc0febf0 2023-07-14T19:02:58.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.356+0000 7f3bb3fff700 1 -- 172.21.15.78:0/1185364340 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+260123 (secure 0 0 0) 0x7f3bc0016090 con 0x7f3bcc0febf0 2023-07-14T19:02:58.363 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:02:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3bb40a11e0 msgr2=0x7f3bb40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:58.364 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3bb40a11e0 0x7f3bb40a3690 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00b5c0 tx=0x7f3bbc005fb0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc0febf0 msgr2=0x7f3bcc19c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:02:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc0febf0 0x7f3bcc19c010 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f3bc000d950 tx=0x7f3bc000dc60 comp rx=0 tx=0).stop 2023-07-14T19:02:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 shutdown_connections 2023-07-14T19:02:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3bb40a11e0 0x7f3bb40a3690 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3bcc0fe240 0x7f3bcc19bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bcc104ca0 0x7f3bcc195b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 --2- 172.21.15.78:0/1185364340 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3bcc0febf0 0x7f3bcc19c010 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:02:58.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 >> 172.21.15.78:0/1185364340 conn(0x7f3bcc075050 msgr2=0x7f3bcc0fbfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:02:58.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 shutdown_connections 2023-07-14T19:02:58.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:02:58.359+0000 7f3bd12c8700 1 -- 172.21.15.78:0/1185364340 wait complete. 2023-07-14T19:02:58.751 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":17,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6800","nonce":2266091126},{"type":"v1","addr":"172.21.15.78:6801","nonce":2266091126}]},"active_addr":"172.21.15.78:6801/2266091126","active_change":"2023-07-14T18:56:30.036967+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24119,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_password":{"name":"alertmanager_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web password","long_desc":"","tags":[],"see_also":[]},"alertmanager_web_user":{"name":"alertmanager_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Alertmanager web user","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"prometheus_web_password":{"name":"prometheus_web_password","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web password","long_desc":"","tags":[],"see_also":[]},"prometheus_web_user":{"name":"prometheus_web_user","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"Prometheus web user","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.78:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":672434764}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":3823618150}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":699667093}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":887313716}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.78:0","nonce":2227474944}]}]} 2023-07-14T19:02:58.758 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-07-14T19:02:58.758 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-07-14T19:02:58.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:02:59.310 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:02:59.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:59 smithi088 ceph-mon[128973]: pgmap v269: 129 pgs: 129 active+clean; 619 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2023-07-14T19:02:59.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:02:59 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1185364340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:02:59.536 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:59 smithi078 ceph-mon[127425]: pgmap v269: 129 pgs: 129 active+clean; 619 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2023-07-14T19:02:59.537 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:02:59 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1185364340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:02:59.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:59 smithi102 ceph-mon[129280]: pgmap v269: 129 pgs: 129 active+clean; 619 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 5.7 KiB/s wr, 12 op/s 2023-07-14T19:02:59.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:02:59 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1185364340' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-07-14T19:03:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.626+0000 7f715a0fd700 1 -- 172.21.15.78:0/1722967348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 msgr2=0x7f71541012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.626+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1722967348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 0x7f71541012f0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7144009990 tx=0x7f7144009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 -- 172.21.15.78:0/1722967348 shutdown_connections 2023-07-14T19:03:00.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1722967348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f715410a3a0 0x7f715410c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1722967348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 0x7f71541012f0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1722967348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f71541018c0 0x7f7154109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 -- 172.21.15.78:0/1722967348 >> 172.21.15.78:0/1722967348 conn(0x7f7154075c90 msgr2=0x7f7154076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 -- 172.21.15.78:0/1722967348 shutdown_connections 2023-07-14T19:03:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 -- 172.21.15.78:0/1722967348 wait complete. 2023-07-14T19:03:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 Processor -- start 2023-07-14T19:03:00.632 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.627+0000 7f715a0fd700 1 -- start start 2023-07-14T19:03:00.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f715a0fd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 0x7f7154195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:00.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f715a0fd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71541018c0 0x7f7154196320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:00.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f715a0fd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f715410a3a0 0x7f715419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:00.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f715a0fd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715419ada0 con 0x7f7154100f20 2023-07-14T19:03:00.634 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f715a0fd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715419af10 con 0x7f715410a3a0 2023-07-14T19:03:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f715a0fd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f715419b0d0 con 0x7f71541018c0 2023-07-14T19:03:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 0x7f7154195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 0x7f7154195de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35080/0 (socket says 172.21.15.78:35080) 2023-07-14T19:03:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 -- 172.21.15.78:0/1174737715 learned_addr learned my addr 172.21.15.78:0/1174737715 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f7153fff700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f715410a3a0 0x7f715419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f7152ffd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71541018c0 0x7f7154196320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:00.635 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 -- 172.21.15.78:0/1174737715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71541018c0 msgr2=0x7f7154196320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71541018c0 0x7f7154196320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 -- 172.21.15.78:0/1174737715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f715410a3a0 msgr2=0x7f715419a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f715410a3a0 0x7f715419a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 -- 172.21.15.78:0/1174737715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71440095f0 con 0x7f7154100f20 2023-07-14T19:03:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f7152ffd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71541018c0 0x7f7154196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f71537fe700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 0x7f7154195de0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7144006010 tx=0x7f71440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:00.636 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f7150ff9700 1 -- 172.21.15.78:0/1174737715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714401d070 con 0x7f7154100f20 2023-07-14T19:03:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f7150ff9700 1 -- 172.21.15.78:0/1174737715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7144004100 con 0x7f7154100f20 2023-07-14T19:03:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f715419b350 con 0x7f7154100f20 2023-07-14T19:03:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f7150ff9700 1 -- 172.21.15.78:0/1174737715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7144022cd0 con 0x7f7154100f20 2023-07-14T19:03:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.628+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71541a2c10 con 0x7f7154100f20 2023-07-14T19:03:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.629+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f715404e340 con 0x7f7154100f20 2023-07-14T19:03:00.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.630+0000 7f7150ff9700 1 -- 172.21.15.78:0/1174737715 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7144022470 con 0x7f7154100f20 2023-07-14T19:03:00.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.631+0000 7f7150ff9700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f713c0a11e0 0x7f713c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:00.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.631+0000 7f7152ffd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f713c0a11e0 0x7f713c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:00.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.631+0000 7f7150ff9700 1 -- 172.21.15.78:0/1174737715 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f71440ce2a0 con 0x7f7154100f20 2023-07-14T19:03:00.638 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.631+0000 7f7152ffd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f713c0a11e0 0x7f713c0a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7148005fd0 tx=0x7f7148005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:00.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.635+0000 7f7150ff9700 1 -- 172.21.15.78:0/1174737715 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f71440976d0 con 0x7f7154100f20 2023-07-14T19:03:00.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.823+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f71541971e0 con 0x7f7154100f20 2023-07-14T19:03:00.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.825+0000 7f7150ff9700 1 -- 172.21.15.78:0/1174737715 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7f714409acf0 con 0x7f7154100f20 2023-07-14T19:03:00.829 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:00.829 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:03:00.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f713c0a11e0 msgr2=0x7f713c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:00.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f713c0a11e0 0x7f713c0a3690 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7148005fd0 tx=0x7f7148005e20 comp rx=0 tx=0).stop 2023-07-14T19:03:00.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 msgr2=0x7f7154195de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:00.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 0x7f7154195de0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7144006010 tx=0x7f71440057f0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 shutdown_connections 2023-07-14T19:03:00.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f713c0a11e0 0x7f713c0a3690 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f715410a3a0 0x7f715419a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7154100f20 0x7f7154195de0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 --2- 172.21.15.78:0/1174737715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f71541018c0 0x7f7154196320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:00.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 >> 172.21.15.78:0/1174737715 conn(0x7f7154075c90 msgr2=0x7f715410add0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:00.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 shutdown_connections 2023-07-14T19:03:00.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:00.827+0000 7f715a0fd700 1 -- 172.21.15.78:0/1174737715 wait complete. 2023-07-14T19:03:01.157 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-07-14T19:03:01.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd dump --format=json 2023-07-14T19:03:01.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:01 smithi088 ceph-mon[128973]: pgmap v270: 129 pgs: 129 active+clean; 623 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.1 KiB/s wr, 14 op/s 2023-07-14T19:03:01.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:01 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1174737715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:01.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:01 smithi078 ceph-mon[127425]: pgmap v270: 129 pgs: 129 active+clean; 623 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.1 KiB/s wr, 14 op/s 2023-07-14T19:03:01.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:01 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1174737715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:01.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:01 smithi102 ceph-mon[129280]: pgmap v270: 129 pgs: 129 active+clean; 623 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 6.1 KiB/s wr, 14 op/s 2023-07-14T19:03:01.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:01 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1174737715' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:01.710 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:03.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 -- 172.21.15.78:0/252473628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fe4b0 msgr2=0x7f4de00fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:03.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 --2- 172.21.15.78:0/252473628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fe4b0 0x7f4de00fe880 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4dd00099c0 tx=0x7f4dd0009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 -- 172.21.15.78:0/252473628 shutdown_connections 2023-07-14T19:03:03.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 --2- 172.21.15.78:0/252473628 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4de0105010 0x7f4de010cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 --2- 172.21.15.78:0/252473628 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fe4b0 0x7f4de00fe880 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 --2- 172.21.15.78:0/252473628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de00fee50 0x7f4de0104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 -- 172.21.15.78:0/252473628 >> 172.21.15.78:0/252473628 conn(0x7f4de0075280 msgr2=0x7f4de0075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:03.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 -- 172.21.15.78:0/252473628 shutdown_connections 2023-07-14T19:03:03.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.031+0000 7f4de502c700 1 -- 172.21.15.78:0/252473628 wait complete. 2023-07-14T19:03:03.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.032+0000 7f4de502c700 1 Processor -- start 2023-07-14T19:03:03.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4de502c700 1 -- start start 2023-07-14T19:03:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4de502c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4de00fe4b0 0x7f4de019ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4de502c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fee50 0x7f4de019bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4de502c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de0105010 0x7f4de0195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4de502c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de0196110 con 0x7f4de00fee50 2023-07-14T19:03:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4de502c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de0196280 con 0x7f4de00fe4b0 2023-07-14T19:03:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4de502c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4de0196440 con 0x7f4de0105010 2023-07-14T19:03:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dde59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fee50 0x7f4de019bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dde59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fee50 0x7f4de019bf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35096/0 (socket says 172.21.15.78:35096) 2023-07-14T19:03:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dde59c700 1 -- 172.21.15.78:0/3214905501 learned_addr learned my addr 172.21.15.78:0/3214905501 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dded9d700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4de00fe4b0 0x7f4de019ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dde59c700 1 -- 172.21.15.78:0/3214905501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de0105010 msgr2=0x7f4de0195aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dde59c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de0105010 0x7f4de0195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dde59c700 1 -- 172.21.15.78:0/3214905501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4de00fe4b0 msgr2=0x7f4de019ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dde59c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4de00fe4b0 0x7f4de019ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.033+0000 7f4dde59c700 1 -- 172.21.15.78:0/3214905501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd00095f0 con 0x7f4de00fee50 2023-07-14T19:03:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.034+0000 7f4dde59c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fee50 0x7f4de019bf60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400b530 tx=0x7f4dd400b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.034+0000 7f4dc7fff700 1 -- 172.21.15.78:0/3214905501 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd4011830 con 0x7f4de00fee50 2023-07-14T19:03:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.034+0000 7f4dc7fff700 1 -- 172.21.15.78:0/3214905501 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4dd4011e20 con 0x7f4de00fee50 2023-07-14T19:03:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.034+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4de0196720 con 0x7f4de00fee50 2023-07-14T19:03:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.034+0000 7f4dc7fff700 1 -- 172.21.15.78:0/3214905501 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dd400f2e0 con 0x7f4de00fee50 2023-07-14T19:03:03.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.034+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4de01a2a10 con 0x7f4de00fee50 2023-07-14T19:03:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.035+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4de004e340 con 0x7f4de00fee50 2023-07-14T19:03:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.036+0000 7f4dc7fff700 1 -- 172.21.15.78:0/3214905501 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4dd4011990 con 0x7f4de00fee50 2023-07-14T19:03:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.036+0000 7f4dc7fff700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4dc80a11e0 0x7f4dc80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.037+0000 7f4dded9d700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4dc80a11e0 0x7f4dc80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.037+0000 7f4dc7fff700 1 -- 172.21.15.78:0/3214905501 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4dd40ccb10 con 0x7f4de00fee50 2023-07-14T19:03:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.037+0000 7f4dded9d700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4dc80a11e0 0x7f4dc80a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000b580 tx=0x7f4dd0009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:03.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.040+0000 7f4dc7fff700 1 -- 172.21.15.78:0/3214905501 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4dd4096020 con 0x7f4de00fee50 2023-07-14T19:03:03.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.228+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4de0196fb0 con 0x7f4de00fee50 2023-07-14T19:03:03.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.229+0000 7f4dc7fff700 1 -- 172.21.15.78:0/3214905501 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+23922 (secure 0 0 0) 0x7f4dd4099640 con 0x7f4de00fee50 2023-07-14T19:03:03.233 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:03.233 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":85,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:02:34.976449+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:03:03.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.231+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4dc80a11e0 msgr2=0x7f4dc80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:03.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.231+0000 7f4de502c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4dc80a11e0 0x7f4dc80a3690 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4dd000b580 tx=0x7f4dd0009f10 comp rx=0 tx=0).stop 2023-07-14T19:03:03.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.231+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fee50 msgr2=0x7f4de019bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:03.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.231+0000 7f4de502c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fee50 0x7f4de019bf60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4dd400b530 tx=0x7f4dd400b840 comp rx=0 tx=0).stop 2023-07-14T19:03:03.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.231+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 shutdown_connections 2023-07-14T19:03:03.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.231+0000 7f4de502c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4dc80a11e0 0x7f4dc80a3690 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.231+0000 7f4de502c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4de00fe4b0 0x7f4de019ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.232+0000 7f4de502c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4de0105010 0x7f4de0195aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.232+0000 7f4de502c700 1 --2- 172.21.15.78:0/3214905501 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4de00fee50 0x7f4de019bf60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:03.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.232+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 >> 172.21.15.78:0/3214905501 conn(0x7f4de0075280 msgr2=0x7f4de0103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:03.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.232+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 shutdown_connections 2023-07-14T19:03:03.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:03.232+0000 7f4de502c700 1 -- 172.21.15.78:0/3214905501 wait complete. 2023-07-14T19:03:03.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:03 smithi088 ceph-mon[128973]: pgmap v271: 129 pgs: 129 active+clean; 623 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2023-07-14T19:03:03.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:03 smithi078 ceph-mon[127425]: pgmap v271: 129 pgs: 129 active+clean; 623 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2023-07-14T19:03:03.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:03 smithi102 ceph-mon[129280]: pgmap v271: 129 pgs: 129 active+clean; 623 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 5.8 KiB/s wr, 14 op/s 2023-07-14T19:03:03.673 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2023-07-14T19:03:03.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2023-07-14T19:03:03.674 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2023-07-14T19:03:03.675 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2023-07-14T19:03:03.675 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2023-07-14T19:03:03.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2023-07-14T19:03:03.676 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2023-07-14T19:03:03.677 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2023-07-14T19:03:03.678 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.8 flush_pg_stats 2023-07-14T19:03:03.678 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.9 flush_pg_stats 2023-07-14T19:03:03.679 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.10 flush_pg_stats 2023-07-14T19:03:03.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph tell osd.11 flush_pg_stats 2023-07-14T19:03:04.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:04 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3214905501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:04.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:04 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3214905501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:04.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:04 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3214905501' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:05.291 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:05.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:05 smithi088 ceph-mon[128973]: pgmap v272: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2023-07-14T19:03:05.530 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:05 smithi078 ceph-mon[127425]: pgmap v272: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2023-07-14T19:03:05.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:05 smithi102 ceph-mon[129280]: pgmap v272: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 8.0 KiB/s rd, 5.2 KiB/s wr, 11 op/s 2023-07-14T19:03:05.755 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:06.007 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:06.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 -- 172.21.15.78:0/2227217327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 msgr2=0x7f3afc0fe680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:06.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 --2- 172.21.15.78:0/2227217327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 0x7f3afc0fe680 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3af0009990 tx=0x7f3af0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:06.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 -- 172.21.15.78:0/2227217327 shutdown_connections 2023-07-14T19:03:06.084 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 --2- 172.21.15.78:0/2227217327 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3afc0febc0 0x7f3afc10cb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:06.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 --2- 172.21.15.78:0/2227217327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3afc104c70 0x7f3afc105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:06.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 --2- 172.21.15.78:0/2227217327 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 0x7f3afc0fe680 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:06.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 -- 172.21.15.78:0/2227217327 >> 172.21.15.78:0/2227217327 conn(0x7f3afc075280 msgr2=0x7f3afc075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:06.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 -- 172.21.15.78:0/2227217327 shutdown_connections 2023-07-14T19:03:06.085 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.080+0000 7f3b0157a700 1 -- 172.21.15.78:0/2227217327 wait complete. 2023-07-14T19:03:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3b0157a700 1 Processor -- start 2023-07-14T19:03:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3b0157a700 1 -- start start 2023-07-14T19:03:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3b0157a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 0x7f3afc195cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3b0157a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3afc0febc0 0x7f3afc196200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3b0157a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3afc104c70 0x7f3afc1995c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:06.087 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3b0157a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc199bc0 con 0x7f3afc0fe210 2023-07-14T19:03:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3b0157a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc199d30 con 0x7f3afc0febc0 2023-07-14T19:03:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3b0157a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3afc199ef0 con 0x7f3afc104c70 2023-07-14T19:03:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3afaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 0x7f3afc195cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3afaffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 0x7f3afc195cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35104/0 (socket says 172.21.15.78:35104) 2023-07-14T19:03:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.081+0000 7f3afaffd700 1 -- 172.21.15.78:0/832624873 learned_addr learned my addr 172.21.15.78:0/832624873 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:06.088 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afa7fc700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3afc0febc0 0x7f3afc196200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:06.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afb7fe700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3afc104c70 0x7f3afc1995c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:06.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afaffd700 1 -- 172.21.15.78:0/832624873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3afc104c70 msgr2=0x7f3afc1995c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:06.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afaffd700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3afc104c70 0x7f3afc1995c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:06.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afaffd700 1 -- 172.21.15.78:0/832624873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3afc0febc0 msgr2=0x7f3afc196200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:06.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afaffd700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3afc0febc0 0x7f3afc196200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:06.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afaffd700 1 -- 172.21.15.78:0/832624873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af00095f0 con 0x7f3afc0fe210 2023-07-14T19:03:06.089 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afb7fe700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3afc104c70 0x7f3afc1995c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:06.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3afaffd700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 0x7f3afc195cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00e9c0 tx=0x7f3aec00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:06.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec0097b0 con 0x7f3afc0fe210 2023-07-14T19:03:06.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3afc19a1d0 con 0x7f3afc0fe210 2023-07-14T19:03:06.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3aec004cc0 con 0x7f3afc0fe210 2023-07-14T19:03:06.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aec005530 con 0x7f3afc0fe210 2023-07-14T19:03:06.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.082+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3afc1a2b20 con 0x7f3afc0fe210 2023-07-14T19:03:06.090 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.083+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3afc10fec0 con 0x7f3afc0fe210 2023-07-14T19:03:06.091 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.084+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3aec004750 con 0x7f3afc0fe210 2023-07-14T19:03:06.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.085+0000 7f3ae3fff700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3ae40a11e0 0x7f3ae40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:06.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.085+0000 7f3afa7fc700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3ae40a11e0 0x7f3ae40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:06.092 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.085+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f3aec014070 con 0x7f3afc0fe210 2023-07-14T19:03:06.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.085+0000 7f3ae3fff700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] conn(0x7f3ae40abd50 0x7f3ae40ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:06.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.085+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 --> [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ae40ae850 con 0x7f3ae40abd50 2023-07-14T19:03:06.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.085+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3aec0cd630 con 0x7f3afc0fe210 2023-07-14T19:03:06.093 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.085+0000 7f3afa7fc700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3ae40a11e0 0x7f3ae40a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3afc197340 tx=0x7f3af000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:06.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.085+0000 7f3afb7fe700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] conn(0x7f3ae40abd50 0x7f3ae40ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:06.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.086+0000 7f3afb7fe700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] conn(0x7f3ae40abd50 0x7f3ae40ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:06.094 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.086+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 <== osd.1 v2:172.21.15.88:6800/3959843234 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f3ae40ae850 con 0x7f3ae40abd50 2023-07-14T19:03:06.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.111+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 --> [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3afc04e340 con 0x7f3ae40abd50 2023-07-14T19:03:06.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3ae3fff700 1 -- 172.21.15.78:0/832624873 <== osd.1 v2:172.21.15.88:6800/3959843234 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3afc04e340 con 0x7f3ae40abd50 2023-07-14T19:03:06.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 >> [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] conn(0x7f3ae40abd50 msgr2=0x7f3ae40ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:06.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] conn(0x7f3ae40abd50 0x7f3ae40ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:06.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3ae40a11e0 msgr2=0x7f3ae40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:06.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3ae40a11e0 0x7f3ae40a3690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3afc197340 tx=0x7f3af000b260 comp rx=0 tx=0).stop 2023-07-14T19:03:06.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 msgr2=0x7f3afc195cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:06.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 --2- 172.21.15.78:0/832624873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3afc0fe210 0x7f3afc195cc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3aec00e9c0 tx=0x7f3aec00ecd0 comp rx=0 tx=0).stop 2023-07-14T19:03:06.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3afb7fe700 1 -- 172.21.15.78:0/832624873 reap_dead start 2023-07-14T19:03:06.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 shutdown_connections 2023-07-14T19:03:06.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 >> 172.21.15.78:0/832624873 conn(0x7f3afc075280 msgr2=0x7f3afc0fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:06.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 shutdown_connections 2023-07-14T19:03:06.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:06.112+0000 7f3b0157a700 1 -- 172.21.15.78:0/832624873 wait complete. 2023-07-14T19:03:06.133 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:06.291 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:06.875 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:06 smithi078 ceph-mon[127425]: pgmap v273: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-07-14T19:03:06.880 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:06 smithi102 ceph-mon[129280]: pgmap v273: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-07-14T19:03:06.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:06 smithi088 ceph-mon[128973]: pgmap v273: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 2.1 KiB/s wr, 7 op/s 2023-07-14T19:03:06.962 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:06.977 INFO:teuthology.orchestra.run.smithi078.stdout:55834574908 2023-07-14T19:03:06.978 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.1 2023-07-14T19:03:06.997 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:07.101 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:08.548 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:08.552 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:08.599 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:09.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:09 smithi078 ceph-mon[127425]: pgmap v274: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-07-14T19:03:09.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:09 smithi088 ceph-mon[128973]: pgmap v274: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-07-14T19:03:09.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:09 smithi102 ceph-mon[129280]: pgmap v274: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2023-07-14T19:03:10.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.562+0000 7fc3bf764700 1 -- 172.21.15.78:0/3020089213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b8103ee0 msgr2=0x7fc3b810ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:10.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.562+0000 7fc3bf764700 1 --2- 172.21.15.78:0/3020089213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b8103ee0 0x7fc3b810ba60 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000b0d0 tx=0x7fc3b000b3e0 comp rx=0 tx=0).stop 2023-07-14T19:03:10.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.562+0000 7fc3bf764700 1 -- 172.21.15.78:0/3020089213 shutdown_connections 2023-07-14T19:03:10.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.562+0000 7fc3bf764700 1 --2- 172.21.15.78:0/3020089213 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b80fee40 0x7fc3b80ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:10.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.562+0000 7fc3bf764700 1 --2- 172.21.15.78:0/3020089213 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b8103ee0 0x7fc3b810ba60 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:10.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.562+0000 7fc3bf764700 1 --2- 172.21.15.78:0/3020089213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3b80ff7e0 0x7fc3b81037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:10.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.562+0000 7fc3bf764700 1 -- 172.21.15.78:0/3020089213 >> 172.21.15.78:0/3020089213 conn(0x7fc3b8075ca0 msgr2=0x7fc3b80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:10.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 -- 172.21.15.78:0/3020089213 shutdown_connections 2023-07-14T19:03:10.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 -- 172.21.15.78:0/3020089213 wait complete. 2023-07-14T19:03:10.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 Processor -- start 2023-07-14T19:03:10.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 -- start start 2023-07-14T19:03:10.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b80fee40 0x7fc3b8100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:10.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b80ff7e0 0x7fc3b8100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:10.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3b8103ee0 0x7fc3b8101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:10.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b81a42f0 con 0x7fc3b80fee40 2023-07-14T19:03:10.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b81a4460 con 0x7fc3b80ff7e0 2023-07-14T19:03:10.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.563+0000 7fc3bf764700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3b81a4620 con 0x7fc3b8103ee0 2023-07-14T19:03:10.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b80fee40 0x7fc3b8100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:10.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b80fee40 0x7fc3b8100850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35120/0 (socket says 172.21.15.78:35120) 2023-07-14T19:03:10.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 -- 172.21.15.78:0/2771448152 learned_addr learned my addr 172.21.15.78:0/2771448152 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:10.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bccff700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b80ff7e0 0x7fc3b8100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:10.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bdd01700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3b8103ee0 0x7fc3b8101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:10.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 -- 172.21.15.78:0/2771448152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3b8103ee0 msgr2=0x7fc3b8101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:10.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3b8103ee0 0x7fc3b8101310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:10.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 -- 172.21.15.78:0/2771448152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b80ff7e0 msgr2=0x7fc3b8100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:10.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3b80ff7e0 0x7fc3b8100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:10.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 -- 172.21.15.78:0/2771448152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b0009d00 con 0x7fc3b80fee40 2023-07-14T19:03:10.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bdd01700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3b8103ee0 0x7fc3b8101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:10.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bd500700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b80fee40 0x7fc3b8100850 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400e9a0 tx=0x7fc3b400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:10.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4009790 con 0x7fc3b80fee40 2023-07-14T19:03:10.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3b81a4900 con 0x7fc3b80fee40 2023-07-14T19:03:10.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc3b400cbd0 con 0x7fc3b80fee40 2023-07-14T19:03:10.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4018680 con 0x7fc3b80fee40 2023-07-14T19:03:10.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.564+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3b81a4e00 con 0x7fc3b80fee40 2023-07-14T19:03:10.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.565+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc3b810eb40 con 0x7fc3b80fee40 2023-07-14T19:03:10.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.566+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fc3b400cd40 con 0x7fc3b80fee40 2023-07-14T19:03:10.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.567+0000 7fc3ae7fc700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a11e0 0x7fc3a40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:10.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.567+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fc3b4014070 con 0x7fc3b80fee40 2023-07-14T19:03:10.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.567+0000 7fc3bccff700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a11e0 0x7fc3a40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:10.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.567+0000 7fc3ae7fc700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] conn(0x7fc3a40abd50 0x7fc3a40ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:10.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.567+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 --> [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3a40ae850 con 0x7fc3a40abd50 2023-07-14T19:03:10.574 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.567+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3b40cd750 con 0x7fc3b80fee40 2023-07-14T19:03:10.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.567+0000 7fc3bdd01700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] conn(0x7fc3a40abd50 0x7fc3a40ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:10.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.568+0000 7fc3bccff700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a11e0 0x7fc3a40a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8102040 tx=0x7fc3a800a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:10.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.568+0000 7fc3bdd01700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] conn(0x7fc3a40abd50 0x7fc3a40ae180 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:10.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.569+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 <== osd.3 v2:172.21.15.78:6810/4191618186 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fc3a40ae850 con 0x7fc3a40abd50 2023-07-14T19:03:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.590+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 --> [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc3b804e340 con 0x7fc3a40abd50 2023-07-14T19:03:10.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.590+0000 7fc3ae7fc700 1 -- 172.21.15.78:0/2771448152 <== osd.3 v2:172.21.15.78:6810/4191618186 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc3b804e340 con 0x7fc3a40abd50 2023-07-14T19:03:10.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] conn(0x7fc3a40abd50 msgr2=0x7fc3a40ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:10.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] conn(0x7fc3a40abd50 0x7fc3a40ae180 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:10.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a11e0 msgr2=0x7fc3a40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:10.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a11e0 0x7fc3a40a3690 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8102040 tx=0x7fc3a800a300 comp rx=0 tx=0).stop 2023-07-14T19:03:10.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b80fee40 msgr2=0x7fc3b8100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:10.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 --2- 172.21.15.78:0/2771448152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3b80fee40 0x7fc3b8100850 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fc3b400e9a0 tx=0x7fc3b400ecb0 comp rx=0 tx=0).stop 2023-07-14T19:03:10.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bdd01700 1 -- 172.21.15.78:0/2771448152 reap_dead start 2023-07-14T19:03:10.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 shutdown_connections 2023-07-14T19:03:10.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 >> 172.21.15.78:0/2771448152 conn(0x7fc3b8075ca0 msgr2=0x7fc3b8068200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:10.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 shutdown_connections 2023-07-14T19:03:10.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:10.591+0000 7fc3bf764700 1 -- 172.21.15.78:0/2771448152 wait complete. 2023-07-14T19:03:11.177 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:11 smithi078 ceph-mon[127425]: pgmap v275: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:03:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.210+0000 7f6f2640d700 1 -- 172.21.15.78:0/773815939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f20103ee0 msgr2=0x7f6f2010ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.210+0000 7f6f2640d700 1 --2- 172.21.15.78:0/773815939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f20103ee0 0x7f6f2010ba60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f6f1800b0d0 tx=0x7f6f1800b3e0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.210+0000 7f6f2640d700 1 -- 172.21.15.78:0/773815939 shutdown_connections 2023-07-14T19:03:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.210+0000 7f6f2640d700 1 --2- 172.21.15.78:0/773815939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f20103ee0 0x7f6f2010ba60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.210+0000 7f6f2640d700 1 --2- 172.21.15.78:0/773815939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f200fee40 0x7f6f200ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.210+0000 7f6f2640d700 1 --2- 172.21.15.78:0/773815939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f200ff7e0 0x7f6f201037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.210+0000 7f6f2640d700 1 -- 172.21.15.78:0/773815939 >> 172.21.15.78:0/773815939 conn(0x7f6f20075ca0 msgr2=0x7f6f200760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.211+0000 7f6f2640d700 1 -- 172.21.15.78:0/773815939 shutdown_connections 2023-07-14T19:03:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.211+0000 7f6f2640d700 1 -- 172.21.15.78:0/773815939 wait complete. 2023-07-14T19:03:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.211+0000 7f6f2640d700 1 Processor -- start 2023-07-14T19:03:11.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.211+0000 7f6f2640d700 1 -- start start 2023-07-14T19:03:11.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f2640d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f200fee40 0x7f6f20196c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f2640d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f200ff7e0 0x7f6f201971c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f2640d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f20103ee0 0x7f6f201a1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f2640d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f20197b00 con 0x7f6f200ff7e0 2023-07-14T19:03:11.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f2640d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f201a2240 con 0x7f6f200fee40 2023-07-14T19:03:11.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f2640d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f201a2400 con 0x7f6f20103ee0 2023-07-14T19:03:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f200ff7e0 0x7f6f201971c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f200ff7e0 0x7f6f201971c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35134/0 (socket says 172.21.15.78:35134) 2023-07-14T19:03:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 -- 172.21.15.78:0/851555727 learned_addr learned my addr 172.21.15.78:0/851555727 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f249aa700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f20103ee0 0x7f6f201a1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 -- 172.21.15.78:0/851555727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f20103ee0 msgr2=0x7f6f201a1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f20103ee0 0x7f6f201a1d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 -- 172.21.15.78:0/851555727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f200fee40 msgr2=0x7f6f20196c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6f200fee40 0x7f6f20196c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 -- 172.21.15.78:0/851555727 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f18009d00 con 0x7f6f200ff7e0 2023-07-14T19:03:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1f7fe700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f200ff7e0 0x7f6f201971c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400d950 tx=0x7f6f1400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14004220 con 0x7f6f200ff7e0 2023-07-14T19:03:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6f14007620 con 0x7f6f200ff7e0 2023-07-14T19:03:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f201a26e0 con 0x7f6f200ff7e0 2023-07-14T19:03:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14005530 con 0x7f6f200ff7e0 2023-07-14T19:03:11.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.212+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f201a2be0 con 0x7f6f200ff7e0 2023-07-14T19:03:11.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.213+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6f2010ebf0 con 0x7f6f200ff7e0 2023-07-14T19:03:11.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.214+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6f14004750 con 0x7f6f200ff7e0 2023-07-14T19:03:11.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.215+0000 7f6f1d7fa700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6f080a11e0 0x7f6f080a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.215+0000 7f6f1ffff700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6f080a11e0 0x7f6f080a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.215+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f6f140cd0b0 con 0x7f6f200ff7e0 2023-07-14T19:03:11.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.215+0000 7f6f1d7fa700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] conn(0x7f6f080abd50 0x7f6f080ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.215+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 --> [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f080ae850 con 0x7f6f080abd50 2023-07-14T19:03:11.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.215+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f140cd4c0 con 0x7f6f200ff7e0 2023-07-14T19:03:11.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.215+0000 7f6f1ffff700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6f080a11e0 0x7f6f080a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6f10005950 tx=0x7f6f100058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:11.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.215+0000 7f6f249aa700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] conn(0x7f6f080abd50 0x7f6f080ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.216+0000 7f6f249aa700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] conn(0x7f6f080abd50 0x7f6f080ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:11.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.217+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 <== osd.10 v2:172.21.15.88:6824/2764459881 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f6f080ae850 con 0x7f6f080abd50 2023-07-14T19:03:11.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.232+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 --> [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6f2004e340 con 0x7f6f080abd50 2023-07-14T19:03:11.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f1d7fa700 1 -- 172.21.15.78:0/851555727 <== osd.10 v2:172.21.15.88:6824/2764459881 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6f2004e340 con 0x7f6f080abd50 2023-07-14T19:03:11.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 >> [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] conn(0x7f6f080abd50 msgr2=0x7f6f080ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] conn(0x7f6f080abd50 0x7f6f080ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6f080a11e0 msgr2=0x7f6f080a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6f080a11e0 0x7f6f080a3690 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6f10005950 tx=0x7f6f100058e0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f200ff7e0 msgr2=0x7f6f201971c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 --2- 172.21.15.78:0/851555727 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6f200ff7e0 0x7f6f201971c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f6f1400d950 tx=0x7f6f1400dc60 comp rx=0 tx=0).stop 2023-07-14T19:03:11.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f249aa700 1 -- 172.21.15.78:0/851555727 reap_dead start 2023-07-14T19:03:11.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 shutdown_connections 2023-07-14T19:03:11.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 >> 172.21.15.78:0/851555727 conn(0x7f6f20075ca0 msgr2=0x7f6f200fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:11.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 shutdown_connections 2023-07-14T19:03:11.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.233+0000 7f6f2640d700 1 -- 172.21.15.78:0/851555727 wait complete. 2023-07-14T19:03:11.290 INFO:teuthology.orchestra.run.smithi078.stdout:107374182453 2023-07-14T19:03:11.291 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.3 2023-07-14T19:03:11.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:11 smithi088 ceph-mon[128973]: pgmap v275: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:03:11.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.505+0000 7fbe85d93700 1 -- 172.21.15.78:0/1960281971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 msgr2=0x7fbe80100440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.505+0000 7fbe85d93700 1 --2- 172.21.15.78:0/1960281971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 0x7fbe80100440 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbe70009990 tx=0x7fbe70009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.510 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.505+0000 7fbe85d93700 1 -- 172.21.15.78:0/1960281971 shutdown_connections 2023-07-14T19:03:11.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.505+0000 7fbe85d93700 1 --2- 172.21.15.78:0/1960281971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe80109620 0x7fbe8010ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.505+0000 7fbe85d93700 1 --2- 172.21.15.78:0/1960281971 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 0x7fbe80100440 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.511 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.505+0000 7fbe85d93700 1 --2- 172.21.15.78:0/1960281971 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80100980 0x7fbe801090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.505+0000 7fbe85d93700 1 -- 172.21.15.78:0/1960281971 >> 172.21.15.78:0/1960281971 conn(0x7fbe80074550 msgr2=0x7fbe80074950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:11.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.506+0000 7fbe85d93700 1 -- 172.21.15.78:0/1960281971 shutdown_connections 2023-07-14T19:03:11.512 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.506+0000 7fbe85d93700 1 -- 172.21.15.78:0/1960281971 wait complete. 2023-07-14T19:03:11.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.506+0000 7fbe85d93700 1 Processor -- start 2023-07-14T19:03:11.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe85d93700 1 -- start start 2023-07-14T19:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe85d93700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 0x7fbe8019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe85d93700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80100980 0x7fbe8019ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe85d93700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe80109620 0x7fbe8019b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe85d93700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8019bd20 con 0x7fbe80100070 2023-07-14T19:03:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe85d93700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8019be90 con 0x7fbe80100980 2023-07-14T19:03:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe85d93700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8019c050 con 0x7fbe80109620 2023-07-14T19:03:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe7f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 0x7fbe8019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe7f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 0x7fbe8019a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35150/0 (socket says 172.21.15.78:35150) 2023-07-14T19:03:11.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.508+0000 7fbe7f7fe700 1 -- 172.21.15.78:0/1374927071 learned_addr learned my addr 172.21.15.78:0/1374927071 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:11.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7effd700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80100980 0x7fbe8019ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7ffff700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe80109620 0x7fbe8019b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7f7fe700 1 -- 172.21.15.78:0/1374927071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe80109620 msgr2=0x7fbe8019b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7f7fe700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbe80109620 0x7fbe8019b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7f7fe700 1 -- 172.21.15.78:0/1374927071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80100980 msgr2=0x7fbe8019ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7f7fe700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbe80100980 0x7fbe8019ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7f7fe700 1 -- 172.21.15.78:0/1374927071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe700095f0 con 0x7fbe80100070 2023-07-14T19:03:11.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7f7fe700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 0x7fbe8019a940 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbe700036c0 tx=0x7fbe700057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:11.519 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7001d070 con 0x7fbe80100070 2023-07-14T19:03:11.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbe70003fc0 con 0x7fbe80100070 2023-07-14T19:03:11.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe8019c2d0 con 0x7fbe80100070 2023-07-14T19:03:11.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe70022ca0 con 0x7fbe80100070 2023-07-14T19:03:11.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.509+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe801a2a20 con 0x7fbe80100070 2023-07-14T19:03:11.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.510+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbe8004e340 con 0x7fbe80100070 2023-07-14T19:03:11.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.511+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbe70022470 con 0x7fbe80100070 2023-07-14T19:03:11.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.512+0000 7fbe7cff9700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbe680a11e0 0x7fbe680a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.512+0000 7fbe7effd700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbe680a11e0 0x7fbe680a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.512+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbe700cec70 con 0x7fbe80100070 2023-07-14T19:03:11.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.512+0000 7fbe7cff9700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] conn(0x7fbe680abd50 0x7fbe680ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:11.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.512+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 --> [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbe680ae850 con 0x7fbe680abd50 2023-07-14T19:03:11.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.512+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe700cefe0 con 0x7fbe80100070 2023-07-14T19:03:11.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.513+0000 7fbe7effd700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbe680a11e0 0x7fbe680a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbe800ffd30 tx=0x7fbe7400b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:11.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.513+0000 7fbe7ffff700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] conn(0x7fbe680abd50 0x7fbe680ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:11.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.513+0000 7fbe7ffff700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] conn(0x7fbe680abd50 0x7fbe680ae180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:11.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.514+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 <== osd.2 v2:172.21.15.102:6800/861816843 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fbe680ae850 con 0x7fbe680abd50 2023-07-14T19:03:11.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.529+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 --> [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fbe8010ed40 con 0x7fbe680abd50 2023-07-14T19:03:11.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.530+0000 7fbe7cff9700 1 -- 172.21.15.78:0/1374927071 <== osd.2 v2:172.21.15.102:6800/861816843 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fbe78007bf0 con 0x7fbe680abd50 2023-07-14T19:03:11.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.530+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 >> [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] conn(0x7fbe680abd50 msgr2=0x7fbe680ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.530+0000 7fbe85d93700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] conn(0x7fbe680abd50 0x7fbe680ae180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.530+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbe680a11e0 msgr2=0x7fbe680a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.530+0000 7fbe85d93700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbe680a11e0 0x7fbe680a3690 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbe800ffd30 tx=0x7fbe7400b5b0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.530+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 msgr2=0x7fbe8019a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:11.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.530+0000 7fbe85d93700 1 --2- 172.21.15.78:0/1374927071 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbe80100070 0x7fbe8019a940 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fbe700036c0 tx=0x7fbe700057f0 comp rx=0 tx=0).stop 2023-07-14T19:03:11.536 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.530+0000 7fbe7ffff700 1 -- 172.21.15.78:0/1374927071 reap_dead start 2023-07-14T19:03:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.531+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 shutdown_connections 2023-07-14T19:03:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.531+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 >> 172.21.15.78:0/1374927071 conn(0x7fbe80074550 msgr2=0x7fbe800fbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.531+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 shutdown_connections 2023-07-14T19:03:11.537 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.531+0000 7fbe85d93700 1 -- 172.21.15.78:0/1374927071 wait complete. 2023-07-14T19:03:11.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:11 smithi102 ceph-mon[129280]: pgmap v275: 129 pgs: 129 active+clean; 635 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2023-07-14T19:03:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 -- 172.21.15.78:0/544305985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ff360 msgr2=0x7f46380ff7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 --2- 172.21.15.78:0/544305985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ff360 0x7f46380ff7d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f46280099c0 tx=0x7f4628009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 -- 172.21.15.78:0/544305985 shutdown_connections 2023-07-14T19:03:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 --2- 172.21.15.78:0/544305985 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380ffd10 0x7f463810ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 --2- 172.21.15.78:0/544305985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4638103ba0 0x7f4638103f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 --2- 172.21.15.78:0/544305985 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ff360 0x7f46380ff7d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 -- 172.21.15.78:0/544305985 >> 172.21.15.78:0/544305985 conn(0x7f4638075280 msgr2=0x7f4638075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 -- 172.21.15.78:0/544305985 shutdown_connections 2023-07-14T19:03:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.639+0000 7f463f62a700 1 -- 172.21.15.78:0/544305985 wait complete. 2023-07-14T19:03:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.640+0000 7f463f62a700 1 Processor -- start 2023-07-14T19:03:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.640+0000 7f463f62a700 1 -- start start 2023-07-14T19:03:12.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.640+0000 7f463f62a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380ff360 0x7f46381008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.640+0000 7f463f62a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ffd10 0x7f4638100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.640+0000 7f463f62a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4638103ba0 0x7f46381a3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.640+0000 7f463f62a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46381016d0 con 0x7f46380ffd10 2023-07-14T19:03:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.640+0000 7f463f62a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46381a4310 con 0x7f46380ff360 2023-07-14T19:03:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.640+0000 7f463f62a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46381a4480 con 0x7f4638103ba0 2023-07-14T19:03:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ffd10 0x7f4638100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ffd10 0x7f4638100de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35166/0 (socket says 172.21.15.78:35166) 2023-07-14T19:03:12.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 -- 172.21.15.78:0/2730575786 learned_addr learned my addr 172.21.15.78:0/2730575786 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463d3c6700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380ff360 0x7f46381008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463dbc7700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4638103ba0 0x7f46381a3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 -- 172.21.15.78:0/2730575786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4638103ba0 msgr2=0x7f46381a3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4638103ba0 0x7f46381a3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 -- 172.21.15.78:0/2730575786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380ff360 msgr2=0x7f46381008a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f46380ff360 0x7f46381008a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 -- 172.21.15.78:0/2730575786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280095f0 con 0x7f46380ffd10 2023-07-14T19:03:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463cbc5700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ffd10 0x7f4638100de0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4628009f50 tx=0x7f4628004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f462801d070 con 0x7f46380ffd10 2023-07-14T19:03:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4628003fc0 con 0x7f46380ffd10 2023-07-14T19:03:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46381a46a0 con 0x7f46380ffd10 2023-07-14T19:03:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4628022c30 con 0x7f46380ffd10 2023-07-14T19:03:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.641+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46381a4b40 con 0x7f46380ffd10 2023-07-14T19:03:12.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.642+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f463804e340 con 0x7f46380ffd10 2023-07-14T19:03:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.644+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4628022470 con 0x7f46380ffd10 2023-07-14T19:03:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.645+0000 7f462e7fc700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f46240a11e0 0x7f46240a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.645+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f46280ce830 con 0x7f46380ffd10 2023-07-14T19:03:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.645+0000 7f463d3c6700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f46240a11e0 0x7f46240a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.645+0000 7f462e7fc700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] conn(0x7f46240abd50 0x7f46240ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.645+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 --> [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f46240ae850 con 0x7f46240abd50 2023-07-14T19:03:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.645+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f46280ceba0 con 0x7f46380ffd10 2023-07-14T19:03:12.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.645+0000 7f463d3c6700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f46240a11e0 0x7f46240a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4634005950 tx=0x7f463400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.645+0000 7f463dbc7700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] conn(0x7f46240abd50 0x7f46240ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.646+0000 7f463dbc7700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] conn(0x7f46240abd50 0x7f46240ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.646+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 <== osd.7 v2:172.21.15.88:6816/1000622113 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f46240ae850 con 0x7f46240abd50 2023-07-14T19:03:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.664+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 --> [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f463810ecc0 con 0x7f46240abd50 2023-07-14T19:03:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.664+0000 7f462e7fc700 1 -- 172.21.15.78:0/2730575786 <== osd.7 v2:172.21.15.88:6816/1000622113 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4630010040 con 0x7f46240abd50 2023-07-14T19:03:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 >> [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] conn(0x7f46240abd50 msgr2=0x7f46240ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] conn(0x7f46240abd50 0x7f46240ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f46240a11e0 msgr2=0x7f46240a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f46240a11e0 0x7f46240a3690 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f4634005950 tx=0x7f463400b410 comp rx=0 tx=0).stop 2023-07-14T19:03:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ffd10 msgr2=0x7f4638100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 --2- 172.21.15.78:0/2730575786 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f46380ffd10 0x7f4638100de0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f4628009f50 tx=0x7f4628004710 comp rx=0 tx=0).stop 2023-07-14T19:03:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463dbc7700 1 -- 172.21.15.78:0/2730575786 reap_dead start 2023-07-14T19:03:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 shutdown_connections 2023-07-14T19:03:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 >> 172.21.15.78:0/2730575786 conn(0x7f4638075280 msgr2=0x7f46380fe210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:12.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 shutdown_connections 2023-07-14T19:03:12.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:11.665+0000 7f463f62a700 1 -- 172.21.15.78:0/2730575786 wait complete. 2023-07-14T19:03:12.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 -- 172.21.15.78:0/3649816370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40fee40 msgr2=0x7f04c40ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 --2- 172.21.15.78:0/3649816370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40fee40 0x7f04c40ff210 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f04ac0099c0 tx=0x7f04ac009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 -- 172.21.15.78:0/3649816370 shutdown_connections 2023-07-14T19:03:12.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 --2- 172.21.15.78:0/3649816370 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04c4103ee0 0x7f04c410ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.948 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 --2- 172.21.15.78:0/3649816370 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40fee40 0x7f04c40ff210 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 --2- 172.21.15.78:0/3649816370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04c40ff7e0 0x7f04c41037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 -- 172.21.15.78:0/3649816370 >> 172.21.15.78:0/3649816370 conn(0x7f04c4075ca0 msgr2=0x7f04c40760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:12.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 -- 172.21.15.78:0/3649816370 shutdown_connections 2023-07-14T19:03:12.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.710+0000 7f04c9b57700 1 -- 172.21.15.78:0/3649816370 wait complete. 2023-07-14T19:03:12.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c9b57700 1 Processor -- start 2023-07-14T19:03:12.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c9b57700 1 -- start start 2023-07-14T19:03:12.949 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c9b57700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04c40fee40 0x7f04c4196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c9b57700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40ff7e0 0x7f04c4197040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c9b57700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04c4103ee0 0x7f04c41a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c9b57700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4197980 con 0x7f04c40ff7e0 2023-07-14T19:03:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c9b57700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c41a20c0 con 0x7f04c40fee40 2023-07-14T19:03:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c9b57700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c41a2280 con 0x7f04c4103ee0 2023-07-14T19:03:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40ff7e0 0x7f04c4197040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c2ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40ff7e0 0x7f04c4197040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35182/0 (socket says 172.21.15.78:35182) 2023-07-14T19:03:12.950 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c2ffd700 1 -- 172.21.15.78:0/3253659136 learned_addr learned my addr 172.21.15.78:0/3253659136 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c37fe700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04c40fee40 0x7f04c4196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c3fff700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04c4103ee0 0x7f04c41a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c2ffd700 1 -- 172.21.15.78:0/3253659136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04c4103ee0 msgr2=0x7f04c41a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c2ffd700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04c4103ee0 0x7f04c41a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c2ffd700 1 -- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04c40fee40 msgr2=0x7f04c4196b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c2ffd700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04c40fee40 0x7f04c4196b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.951 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c2ffd700 1 -- 172.21.15.78:0/3253659136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ac0095f0 con 0x7f04c40ff7e0 2023-07-14T19:03:12.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c3fff700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f04c4103ee0 0x7f04c41a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:12.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.711+0000 7f04c37fe700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f04c40fee40 0x7f04c4196b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:12.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.712+0000 7f04c2ffd700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40ff7e0 0x7f04c4197040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f04b400d950 tx=0x7f04b400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:12.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.712+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b400bb60 con 0x7f04c40ff7e0 2023-07-14T19:03:12.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.712+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f04b4007620 con 0x7f04c40ff7e0 2023-07-14T19:03:12.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.712+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04c41a2560 con 0x7f04c40ff7e0 2023-07-14T19:03:12.952 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.712+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04b40053e0 con 0x7f04c40ff7e0 2023-07-14T19:03:12.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.712+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04c41a2a60 con 0x7f04c40ff7e0 2023-07-14T19:03:12.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.713+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f04c410ec50 con 0x7f04c40ff7e0 2023-07-14T19:03:12.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.714+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f04b4005540 con 0x7f04c40ff7e0 2023-07-14T19:03:12.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.714+0000 7f04c0ff9700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f04b00a11e0 0x7f04b00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.714+0000 7f04c37fe700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f04b00a11e0 0x7f04b00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.715+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f04b40ccd10 con 0x7f04c40ff7e0 2023-07-14T19:03:12.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.715+0000 7f04c0ff9700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] conn(0x7f04b00abd50 0x7f04b00ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:12.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.715+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 --> [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f04b00ae850 con 0x7f04b00abd50 2023-07-14T19:03:12.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.715+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f04b40cd120 con 0x7f04c40ff7e0 2023-07-14T19:03:12.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.715+0000 7f04c37fe700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f04b00a11e0 0x7f04b00a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000c00 tx=0x7f04ac005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:12.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.715+0000 7f04c3fff700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] conn(0x7f04b00abd50 0x7f04b00ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:12.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.715+0000 7f04c3fff700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] conn(0x7f04b00abd50 0x7f04b00ae180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.716+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 <== osd.4 v2:172.21.15.88:6808/1708042388 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f04b00ae850 con 0x7f04b00abd50 2023-07-14T19:03:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.731+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 --> [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f04c404e340 con 0x7f04b00abd50 2023-07-14T19:03:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.732+0000 7f04c0ff9700 1 -- 172.21.15.78:0/3253659136 <== osd.4 v2:172.21.15.88:6808/1708042388 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f04c404e340 con 0x7f04b00abd50 2023-07-14T19:03:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] conn(0x7f04b00abd50 msgr2=0x7f04b00ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] conn(0x7f04b00abd50 0x7f04b00ae180 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f04b00a11e0 msgr2=0x7f04b00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f04b00a11e0 0x7f04b00a3690 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f04ac000c00 tx=0x7f04ac005fb0 comp rx=0 tx=0).stop 2023-07-14T19:03:12.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40ff7e0 msgr2=0x7f04c4197040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:12.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 --2- 172.21.15.78:0/3253659136 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f04c40ff7e0 0x7f04c4197040 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f04b400d950 tx=0x7f04b400dd10 comp rx=0 tx=0).stop 2023-07-14T19:03:12.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c3fff700 1 -- 172.21.15.78:0/3253659136 reap_dead start 2023-07-14T19:03:12.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 shutdown_connections 2023-07-14T19:03:12.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 >> 172.21.15.78:0/3253659136 conn(0x7f04c4075ca0 msgr2=0x7f04c41027f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:12.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 shutdown_connections 2023-07-14T19:03:12.956 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:12.733+0000 7f04c9b57700 1 -- 172.21.15.78:0/3253659136 wait complete. 2023-07-14T19:03:12.958 INFO:teuthology.orchestra.run.smithi078.stdout:77309411386 2023-07-14T19:03:12.959 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.2 2023-07-14T19:03:13.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:13 smithi078 ceph-mon[127425]: pgmap v276: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:03:13.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:13 smithi088 ceph-mon[128973]: pgmap v276: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:03:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 -- 172.21.15.78:0/696415465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4103ee0 msgr2=0x7f1ff410ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 --2- 172.21.15.78:0/696415465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4103ee0 0x7f1ff410ba60 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f1fec00b0d0 tx=0x7f1fec00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:03:13.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 -- 172.21.15.78:0/696415465 shutdown_connections 2023-07-14T19:03:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 --2- 172.21.15.78:0/696415465 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4103ee0 0x7f1ff410ba60 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 --2- 172.21.15.78:0/696415465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ff40fee40 0x7f1ff40ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:13.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 --2- 172.21.15.78:0/696415465 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff40ff7e0 0x7f1ff41037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 -- 172.21.15.78:0/696415465 >> 172.21.15.78:0/696415465 conn(0x7f1ff4075ca0 msgr2=0x7f1ff40760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 -- 172.21.15.78:0/696415465 shutdown_connections 2023-07-14T19:03:13.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 -- 172.21.15.78:0/696415465 wait complete. 2023-07-14T19:03:13.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.292+0000 7f1ffa4af700 1 Processor -- start 2023-07-14T19:03:13.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ffa4af700 1 -- start start 2023-07-14T19:03:13.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ffa4af700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff40fee40 0x7f1ff4196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:13.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ffa4af700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ff40ff7e0 0x7f1ff4197180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:13.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ffa4af700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4103ee0 0x7f1ff41a1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:13.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ffa4af700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff41a2200 con 0x7f1ff4103ee0 2023-07-14T19:03:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ffa4af700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff41a2340 con 0x7f1ff40fee40 2023-07-14T19:03:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ffa4af700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff41a2480 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ff8a4c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4103ee0 0x7f1ff41a1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ff3fff700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff40fee40 0x7f1ff4196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ff8a4c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4103ee0 0x7f1ff41a1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35198/0 (socket says 172.21.15.78:35198) 2023-07-14T19:03:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ff8a4c700 1 -- 172.21.15.78:0/26802714 learned_addr learned my addr 172.21.15.78:0/26802714 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:13.544 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.293+0000 7f1ff37fe700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ff40ff7e0 0x7f1ff4197180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff37fe700 1 -- 172.21.15.78:0/26802714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff40fee40 msgr2=0x7f1ff4196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff37fe700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ff40fee40 0x7f1ff4196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff37fe700 1 -- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4103ee0 msgr2=0x7f1ff41a1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff37fe700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ff4103ee0 0x7f1ff41a1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff37fe700 1 -- 172.21.15.78:0/26802714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec009d00 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff37fe700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ff40ff7e0 0x7f1ff4197180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4069e10 tx=0x7f1fe800dc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe800f890 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.545 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1fe800fe80 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff4194a90 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe800e3b0 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.294+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff4194f90 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.295+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ff410ebb0 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.296+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1fe800f9f0 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff17fa700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1fdc0a11e0 0x7f1fdc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:13.546 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff3fff700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1fdc0a11e0 0x7f1fdc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1fe80cce00 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff17fa700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] conn(0x7f1fdc0abd50 0x7f1fdc0ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 --> [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fdc0ae850 con 0x7f1fdc0abd50 2023-07-14T19:03:13.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fe80cd210 con 0x7f1ff40ff7e0 2023-07-14T19:03:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff8a4c700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] conn(0x7f1fdc0abd50 0x7f1fdc0ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff3fff700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1fdc0a11e0 0x7f1fdc0a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4005950 tx=0x7f1fe4013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.297+0000 7f1ff8a4c700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] conn(0x7f1fdc0abd50 0x7f1fdc0ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.298+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 <== osd.9 v2:172.21.15.78:6826/1815793298 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1fdc0ae850 con 0x7f1fdc0abd50 2023-07-14T19:03:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.313+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 --> [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1ff404e340 con 0x7f1fdc0abd50 2023-07-14T19:03:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ff17fa700 1 -- 172.21.15.78:0/26802714 <== osd.9 v2:172.21.15.78:6826/1815793298 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1ff404e340 con 0x7f1fdc0abd50 2023-07-14T19:03:13.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] conn(0x7f1fdc0abd50 msgr2=0x7f1fdc0ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] conn(0x7f1fdc0abd50 0x7f1fdc0ae180 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1fdc0a11e0 msgr2=0x7f1fdc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1fdc0a11e0 0x7f1fdc0a3690 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4005950 tx=0x7f1fe4013040 comp rx=0 tx=0).stop 2023-07-14T19:03:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ff40ff7e0 msgr2=0x7f1ff4197180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 --2- 172.21.15.78:0/26802714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ff40ff7e0 0x7f1ff4197180 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1ff4069e10 tx=0x7f1fe800dc30 comp rx=0 tx=0).stop 2023-07-14T19:03:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ff8a4c700 1 -- 172.21.15.78:0/26802714 reap_dead start 2023-07-14T19:03:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 shutdown_connections 2023-07-14T19:03:13.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 >> 172.21.15.78:0/26802714 conn(0x7f1ff4075ca0 msgr2=0x7f1ff40fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:13.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 shutdown_connections 2023-07-14T19:03:13.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:13.314+0000 7f1ffa4af700 1 -- 172.21.15.78:0/26802714 wait complete. 2023-07-14T19:03:13.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:13 smithi102 ceph-mon[129280]: pgmap v276: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:03:14.097 INFO:teuthology.orchestra.run.smithi078.stdout:274877906972 2023-07-14T19:03:14.097 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.9 2023-07-14T19:03:14.650 INFO:teuthology.orchestra.run.smithi078.stdout:141733920818 2023-07-14T19:03:14.651 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.4 2023-07-14T19:03:15.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 -- 172.21.15.78:0/637349129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fe5e0 msgr2=0x7f1ae80fe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 --2- 172.21.15.78:0/637349129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fe5e0 0x7f1ae80fe9b0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8009990 tx=0x7f1ad8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 -- 172.21.15.78:0/637349129 shutdown_connections 2023-07-14T19:03:15.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 --2- 172.21.15.78:0/637349129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae8103680 0x7f1ae810b200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 --2- 172.21.15.78:0/637349129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fe5e0 0x7f1ae80fe9b0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 --2- 172.21.15.78:0/637349129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae80fef80 0x7f1ae8102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 -- 172.21.15.78:0/637349129 >> 172.21.15.78:0/637349129 conn(0x7f1ae8075280 msgr2=0x7f1ae8075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 -- 172.21.15.78:0/637349129 shutdown_connections 2023-07-14T19:03:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.013+0000 7f1aed03a700 1 -- 172.21.15.78:0/637349129 wait complete. 2023-07-14T19:03:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1aed03a700 1 Processor -- start 2023-07-14T19:03:15.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1aed03a700 1 -- start start 2023-07-14T19:03:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1aed03a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae80fe5e0 0x7f1ae8195cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1aed03a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fef80 0x7f1ae81961f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1aed03a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8103680 0x7f1ae81995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1aed03a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8199bb0 con 0x7f1ae80fef80 2023-07-14T19:03:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1aed03a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8199d20 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1aed03a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae8199ee0 con 0x7f1ae8103680 2023-07-14T19:03:15.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1ae659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fef80 0x7f1ae81961f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1ae659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fef80 0x7f1ae81961f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35226/0 (socket says 172.21.15.78:35226) 2023-07-14T19:03:15.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1ae659c700 1 -- 172.21.15.78:0/3081932954 learned_addr learned my addr 172.21.15.78:0/3081932954 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:15.076 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.014+0000 7f1ae6d9d700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae80fe5e0 0x7f1ae8195cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1ae759e700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8103680 0x7f1ae81995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1ae6d9d700 1 -- 172.21.15.78:0/3081932954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8103680 msgr2=0x7f1ae81995b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1ae6d9d700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae8103680 0x7f1ae81995b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1ae6d9d700 1 -- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fef80 msgr2=0x7f1ae81961f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1ae6d9d700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fef80 0x7f1ae81961f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1ae6d9d700 1 -- 172.21.15.78:0/3081932954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad80095f0 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.077 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1ae659c700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1ae80fef80 0x7f1ae81961f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:03:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1ae6d9d700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae80fe5e0 0x7f1ae8195cb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8005910 tx=0x7f1ad8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad801d070 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae819a160 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 <== mon.2 v2:172.21.15.88:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1ad8022470 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.078 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ad8017650 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.015+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae819a400 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.016+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ae804e340 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.018+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 <== mon.2 v2:172.21.15.88:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1ad80225e0 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.018+0000 7f1acffff700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1ad00a11e0 0x7f1ad00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.019+0000 7f1ae659c700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1ad00a11e0 0x7f1ad00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.019+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 <== mon.2 v2:172.21.15.88:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1ad80ce1a0 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.019+0000 7f1acffff700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] conn(0x7f1ad00abd50 0x7f1ad00ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.019+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 --> [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1ad00ae850 con 0x7f1ad00abd50 2023-07-14T19:03:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.019+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 <== mon.2 v2:172.21.15.88:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ad80ce5a0 con 0x7f1ae80fe5e0 2023-07-14T19:03:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.019+0000 7f1ae759e700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] conn(0x7f1ad00abd50 0x7f1ad00ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.019+0000 7f1ae659c700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1ad00a11e0 0x7f1ad00a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8197290 tx=0x7f1adc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:15.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.019+0000 7f1ae759e700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] conn(0x7f1ad00abd50 0x7f1ad00ae180 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.020+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 <== osd.0 v2:172.21.15.78:6802/1630210341 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1ad00ae850 con 0x7f1ad00abd50 2023-07-14T19:03:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.041+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 --> [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1ae810fde0 con 0x7f1ad00abd50 2023-07-14T19:03:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1acffff700 1 -- 172.21.15.78:0/3081932954 <== osd.0 v2:172.21.15.78:6802/1630210341 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1ae000f040 con 0x7f1ad00abd50 2023-07-14T19:03:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] conn(0x7f1ad00abd50 msgr2=0x7f1ad00ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] conn(0x7f1ad00abd50 0x7f1ad00ae180 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1ad00a11e0 msgr2=0x7f1ad00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.081 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1ad00a11e0 0x7f1ad00a3690 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8197290 tx=0x7f1adc009260 comp rx=0 tx=0).stop 2023-07-14T19:03:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae80fe5e0 msgr2=0x7f1ae8195cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 --2- 172.21.15.78:0/3081932954 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1ae80fe5e0 0x7f1ae8195cb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1ad8005910 tx=0x7f1ad8004730 comp rx=0 tx=0).stop 2023-07-14T19:03:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1ae759e700 1 -- 172.21.15.78:0/3081932954 reap_dead start 2023-07-14T19:03:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 shutdown_connections 2023-07-14T19:03:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 >> 172.21.15.78:0/3081932954 conn(0x7f1ae8075280 msgr2=0x7f1ae80fbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 shutdown_connections 2023-07-14T19:03:15.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.042+0000 7f1aed03a700 1 -- 172.21.15.78:0/3081932954 wait complete. 2023-07-14T19:03:15.201 INFO:teuthology.orchestra.run.smithi078.stdout:296352743447 2023-07-14T19:03:15.201 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.10 2023-07-14T19:03:15.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 -- 172.21.15.78:0/930652493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 msgr2=0x7f1d68102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 --2- 172.21.15.78:0/930652493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 0x7f1d68102f90 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c0099c0 tx=0x7f1d5c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 -- 172.21.15.78:0/930652493 shutdown_connections 2023-07-14T19:03:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 --2- 172.21.15.78:0/930652493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d681036c0 0x7f1d6810b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 --2- 172.21.15.78:0/930652493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d680fe620 0x7f1d680fe9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 --2- 172.21.15.78:0/930652493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 0x7f1d68102f90 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 -- 172.21.15.78:0/930652493 >> 172.21.15.78:0/930652493 conn(0x7f1d680751e0 msgr2=0x7f1d680755e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 -- 172.21.15.78:0/930652493 shutdown_connections 2023-07-14T19:03:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 -- 172.21.15.78:0/930652493 wait complete. 2023-07-14T19:03:15.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 Processor -- start 2023-07-14T19:03:15.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.536+0000 7f1d6e430700 1 -- start start 2023-07-14T19:03:15.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6e430700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d680fe620 0x7f1d6819bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6e430700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 0x7f1d6819c010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6e430700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d681036c0 0x7f1d68195b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6e430700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d681961c0 con 0x7f1d680fefc0 2023-07-14T19:03:15.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6e430700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d68196330 con 0x7f1d681036c0 2023-07-14T19:03:15.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6e430700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d681964f0 con 0x7f1d680fe620 2023-07-14T19:03:15.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 0x7f1d6819c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 0x7f1d6819c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35210/0 (socket says 172.21.15.78:35210) 2023-07-14T19:03:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 -- 172.21.15.78:0/1051740278 learned_addr learned my addr 172.21.15.78:0/1051740278 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d67fff700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d680fe620 0x7f1d6819bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6c9cd700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d681036c0 0x7f1d68195b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 -- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d680fe620 msgr2=0x7f1d6819bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d680fe620 0x7f1d6819bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 -- 172.21.15.78:0/1051740278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d681036c0 msgr2=0x7f1d68195b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d681036c0 0x7f1d68195b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 -- 172.21.15.78:0/1051740278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d5c0095f0 con 0x7f1d680fefc0 2023-07-14T19:03:15.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d67fff700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1d680fe620 0x7f1d6819bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:15.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6c9cd700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1d681036c0 0x7f1d68195b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:15.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d677fe700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 0x7f1d6819c010 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c006010 tx=0x7f1d5c009510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:15.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c01d070 con 0x7f1d680fefc0 2023-07-14T19:03:15.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d68196770 con 0x7f1d680fefc0 2023-07-14T19:03:15.255 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1d5c004100 con 0x7f1d680fefc0 2023-07-14T19:03:15.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.537+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d5c022d70 con 0x7f1d680fefc0 2023-07-14T19:03:15.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.538+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d681a2ac0 con 0x7f1d680fefc0 2023-07-14T19:03:15.256 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.538+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1d6804e340 con 0x7f1d680fefc0 2023-07-14T19:03:15.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.539+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1d5c022570 con 0x7f1d680fefc0 2023-07-14T19:03:15.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.540+0000 7f1d657fa700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1d500a11e0 0x7f1d500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.540+0000 7f1d67fff700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1d500a11e0 0x7f1d500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.540+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f1d5c0ceab0 con 0x7f1d680fefc0 2023-07-14T19:03:15.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.540+0000 7f1d657fa700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] conn(0x7f1d500abd50 0x7f1d500ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:15.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.540+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 --> [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1d500ae850 con 0x7f1d500abd50 2023-07-14T19:03:15.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.540+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1d5c0ceec0 con 0x7f1d680fefc0 2023-07-14T19:03:15.257 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.540+0000 7f1d67fff700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1d500a11e0 0x7f1d500a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005950 tx=0x7f1d58005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:15.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.540+0000 7f1d6c9cd700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] conn(0x7f1d500abd50 0x7f1d500ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:15.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.541+0000 7f1d6c9cd700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] conn(0x7f1d500abd50 0x7f1d500ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:15.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.542+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 <== osd.8 v2:172.21.15.102:6816/4258575075 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f1d500ae850 con 0x7f1d500abd50 2023-07-14T19:03:15.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.557+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 --> [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1d6810e410 con 0x7f1d500abd50 2023-07-14T19:03:15.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.558+0000 7f1d657fa700 1 -- 172.21.15.78:0/1051740278 <== osd.8 v2:172.21.15.102:6816/4258575075 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1d600097e0 con 0x7f1d500abd50 2023-07-14T19:03:15.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.558+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] conn(0x7f1d500abd50 msgr2=0x7f1d500ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.258 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.558+0000 7f1d6e430700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] conn(0x7f1d500abd50 0x7f1d500ae180 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:15.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.558+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1d500a11e0 msgr2=0x7f1d500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.558+0000 7f1d6e430700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f1d500a11e0 0x7f1d500a3690 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1d58005950 tx=0x7f1d58005f50 comp rx=0 tx=0).stop 2023-07-14T19:03:15.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.558+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 msgr2=0x7f1d6819c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:15.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.558+0000 7f1d6e430700 1 --2- 172.21.15.78:0/1051740278 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1d680fefc0 0x7f1d6819c010 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1d5c006010 tx=0x7f1d5c009510 comp rx=0 tx=0).stop 2023-07-14T19:03:15.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.558+0000 7f1d6c9cd700 1 -- 172.21.15.78:0/1051740278 reap_dead start 2023-07-14T19:03:15.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.559+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 shutdown_connections 2023-07-14T19:03:15.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.559+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 >> 172.21.15.78:0/1051740278 conn(0x7f1d680751e0 msgr2=0x7f1d6810ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:15.259 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.559+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 shutdown_connections 2023-07-14T19:03:15.260 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:14.559+0000 7f1d6e430700 1 -- 172.21.15.78:0/1051740278 wait complete. 2023-07-14T19:03:15.305 INFO:teuthology.orchestra.run.smithi078.stdout:214748364837 2023-07-14T19:03:15.306 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.7 2023-07-14T19:03:15.339 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:15 smithi078 ceph-mon[127425]: pgmap v277: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:03:15.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:15 smithi088 ceph-mon[128973]: pgmap v277: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:03:15.581 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:15.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:15 smithi102 ceph-mon[129280]: pgmap v277: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.6 KiB/s wr, 7 op/s 2023-07-14T19:03:16.089 INFO:teuthology.orchestra.run.smithi078.stdout:34359738435 2023-07-14T19:03:16.089 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.0 2023-07-14T19:03:16.286 INFO:teuthology.orchestra.run.smithi078.stdout:244813135904 2023-07-14T19:03:16.287 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.8 2023-07-14T19:03:16.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 -- 172.21.15.78:0/3547800311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80fee40 msgr2=0x7f95a80ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 --2- 172.21.15.78:0/3547800311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80fee40 0x7f95a80ff210 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f959800b0d0 tx=0x7f959800b3e0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 -- 172.21.15.78:0/3547800311 shutdown_connections 2023-07-14T19:03:16.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 --2- 172.21.15.78:0/3547800311 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95a8103ee0 0x7f95a810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.365 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 --2- 172.21.15.78:0/3547800311 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80fee40 0x7f95a80ff210 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 --2- 172.21.15.78:0/3547800311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95a80ff7e0 0x7f95a81037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 -- 172.21.15.78:0/3547800311 >> 172.21.15.78:0/3547800311 conn(0x7f95a8075ca0 msgr2=0x7f95a80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:16.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 -- 172.21.15.78:0/3547800311 shutdown_connections 2023-07-14T19:03:16.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.709+0000 7f95acdac700 1 -- 172.21.15.78:0/3547800311 wait complete. 2023-07-14T19:03:16.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95acdac700 1 Processor -- start 2023-07-14T19:03:16.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95acdac700 1 -- start start 2023-07-14T19:03:16.366 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95acdac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95a80fee40 0x7f95a8195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95acdac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80ff7e0 0x7f95a81963a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95acdac700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95a8103ee0 0x7f95a819a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95acdac700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819ae20 con 0x7f95a80ff7e0 2023-07-14T19:03:16.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95acdac700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819af90 con 0x7f95a80fee40 2023-07-14T19:03:16.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95acdac700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a819b150 con 0x7f95a8103ee0 2023-07-14T19:03:16.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80ff7e0 0x7f95a81963a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a5d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80ff7e0 0x7f95a81963a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35242/0 (socket says 172.21.15.78:35242) 2023-07-14T19:03:16.367 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a5d9b700 1 -- 172.21.15.78:0/68615352 learned_addr learned my addr 172.21.15.78:0/68615352 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:16.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a6d9d700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95a8103ee0 0x7f95a819a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a659c700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95a80fee40 0x7f95a8195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a5d9b700 1 -- 172.21.15.78:0/68615352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95a8103ee0 msgr2=0x7f95a819a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a5d9b700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95a8103ee0 0x7f95a819a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a5d9b700 1 -- 172.21.15.78:0/68615352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95a80fee40 msgr2=0x7f95a8195e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a5d9b700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f95a80fee40 0x7f95a8195e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.368 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.710+0000 7f95a5d9b700 1 -- 172.21.15.78:0/68615352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598009d00 con 0x7f95a80ff7e0 2023-07-14T19:03:16.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.711+0000 7f95a5d9b700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80ff7e0 0x7f95a81963a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f959000c8f0 tx=0x7f959000cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:16.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.711+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95900078d0 con 0x7f95a80ff7e0 2023-07-14T19:03:16.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.711+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f959000f450 con 0x7f95a80ff7e0 2023-07-14T19:03:16.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.711+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a819b430 con 0x7f95a80ff7e0 2023-07-14T19:03:16.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.711+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9590018660 con 0x7f95a80ff7e0 2023-07-14T19:03:16.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.711+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a81a2c90 con 0x7f95a80ff7e0 2023-07-14T19:03:16.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.712+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f95a810ec50 con 0x7f95a80ff7e0 2023-07-14T19:03:16.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.713+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f959000f5c0 con 0x7f95a80ff7e0 2023-07-14T19:03:16.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.713+0000 7f959f7fe700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95940a11e0 0x7f95940a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.713+0000 7f95a659c700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95940a11e0 0x7f95940a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.713+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f95900ccdb0 con 0x7f95a80ff7e0 2023-07-14T19:03:16.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.714+0000 7f959f7fe700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] conn(0x7f95940abd50 0x7f95940ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.714+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 --> [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f95940ae850 con 0x7f95940abd50 2023-07-14T19:03:16.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.714+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95900cd1c0 con 0x7f95a80ff7e0 2023-07-14T19:03:16.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.714+0000 7f95a6d9d700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] conn(0x7f95940abd50 0x7f95940ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.714+0000 7f95a659c700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95940a11e0 0x7f95940a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9598009fd0 tx=0x7f959800b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:16.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.714+0000 7f95a6d9d700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] conn(0x7f95940abd50 0x7f95940ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:16.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.715+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 <== osd.6 v2:172.21.15.78:6818/2536976719 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f95940ae850 con 0x7f95940abd50 2023-07-14T19:03:16.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 --> [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f95a804e340 con 0x7f95940abd50 2023-07-14T19:03:16.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f959f7fe700 1 -- 172.21.15.78:0/68615352 <== osd.6 v2:172.21.15.78:6818/2536976719 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f95a804e340 con 0x7f95940abd50 2023-07-14T19:03:16.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] conn(0x7f95940abd50 msgr2=0x7f95940ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f95acdac700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] conn(0x7f95940abd50 0x7f95940ae180 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95940a11e0 msgr2=0x7f95940a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f95acdac700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f95940a11e0 0x7f95940a3690 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9598009fd0 tx=0x7f959800b040 comp rx=0 tx=0).stop 2023-07-14T19:03:16.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80ff7e0 msgr2=0x7f95a81963a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f95acdac700 1 --2- 172.21.15.78:0/68615352 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f95a80ff7e0 0x7f95a81963a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f959000c8f0 tx=0x7f959000cc00 comp rx=0 tx=0).stop 2023-07-14T19:03:16.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.730+0000 7f95a6d9d700 1 -- 172.21.15.78:0/68615352 reap_dead start 2023-07-14T19:03:16.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.731+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 shutdown_connections 2023-07-14T19:03:16.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.731+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 >> 172.21.15.78:0/68615352 conn(0x7f95a8075ca0 msgr2=0x7f95a8102af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:16.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.731+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 shutdown_connections 2023-07-14T19:03:16.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:15.731+0000 7f95acdac700 1 -- 172.21.15.78:0/68615352 wait complete. 2023-07-14T19:03:16.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 -- 172.21.15.78:0/874937973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81100ff7e0 msgr2=0x7f81101037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 --2- 172.21.15.78:0/874937973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81100ff7e0 0x7f81101037b0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f80fc0099c0 tx=0x7f80fc009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 -- 172.21.15.78:0/874937973 shutdown_connections 2023-07-14T19:03:16.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 --2- 172.21.15.78:0/874937973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81100fee40 0x7f81100ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 --2- 172.21.15.78:0/874937973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8110103ee0 0x7f811010ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 --2- 172.21.15.78:0/874937973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f81100ff7e0 0x7f81101037b0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 -- 172.21.15.78:0/874937973 >> 172.21.15.78:0/874937973 conn(0x7f8110075ca0 msgr2=0x7f81100760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:16.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 -- 172.21.15.78:0/874937973 shutdown_connections 2023-07-14T19:03:16.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.379+0000 7f8116f32700 1 -- 172.21.15.78:0/874937973 wait complete. 2023-07-14T19:03:16.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8116f32700 1 Processor -- start 2023-07-14T19:03:16.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8116f32700 1 -- start start 2023-07-14T19:03:16.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8116f32700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81100fee40 0x7f8110195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8116f32700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81100ff7e0 0x7f8110196250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8116f32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8110103ee0 0x7f811019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8116f32700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811019acd0 con 0x7f8110103ee0 2023-07-14T19:03:16.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8116f32700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811019ae40 con 0x7f81100fee40 2023-07-14T19:03:16.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8116f32700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811019b000 con 0x7f81100ff7e0 2023-07-14T19:03:16.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f81154cf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8110103ee0 0x7f811019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f81154cf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8110103ee0 0x7f811019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35260/0 (socket says 172.21.15.78:35260) 2023-07-14T19:03:16.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f81154cf700 1 -- 172.21.15.78:0/2601263423 learned_addr learned my addr 172.21.15.78:0/2601263423 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:16.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8114cce700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81100fee40 0x7f8110195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f810ffff700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81100ff7e0 0x7f8110196250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f81154cf700 1 -- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81100ff7e0 msgr2=0x7f8110196250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f81154cf700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81100ff7e0 0x7f8110196250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f81154cf700 1 -- 172.21.15.78:0/2601263423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81100fee40 msgr2=0x7f8110195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f81154cf700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81100fee40 0x7f8110195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f81154cf700 1 -- 172.21.15.78:0/2601263423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc0095f0 con 0x7f8110103ee0 2023-07-14T19:03:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f810ffff700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f81100ff7e0 0x7f8110196250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.380+0000 7f8114cce700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f81100fee40 0x7f8110195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:03:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.381+0000 7f81154cf700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8110103ee0 0x7f811019a6c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f810800efc0 tx=0x7f810800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.381+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108018070 con 0x7f8110103ee0 2023-07-14T19:03:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.381+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811019b280 con 0x7f8110103ee0 2023-07-14T19:03:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.381+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f810800f040 con 0x7f8110103ee0 2023-07-14T19:03:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.381+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810800cbb0 con 0x7f8110103ee0 2023-07-14T19:03:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.381+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811019b530 con 0x7f8110103ee0 2023-07-14T19:03:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.382+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f811004e340 con 0x7f8110103ee0 2023-07-14T19:03:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.384+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f810800cd10 con 0x7f8110103ee0 2023-07-14T19:03:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.384+0000 7f810dffb700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f81000a11e0 0x7f81000a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.384+0000 7f8114cce700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f81000a11e0 0x7f81000a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.384+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f81080cce40 con 0x7f8110103ee0 2023-07-14T19:03:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.385+0000 7f810dffb700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] conn(0x7f81000abd50 0x7f81000ae180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.385+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 --> [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f81000ae850 con 0x7f81000abd50 2023-07-14T19:03:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.385+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81080cd250 con 0x7f8110103ee0 2023-07-14T19:03:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.385+0000 7f8114cce700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f81000a11e0 0x7f81000a3690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8104011490 tx=0x7f8104009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.385+0000 7f810ffff700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] conn(0x7f81000abd50 0x7f81000ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.385+0000 7f810ffff700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] conn(0x7f81000abd50 0x7f81000ae180 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.386+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 <== osd.11 v2:172.21.15.102:6824/383662404 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7f81000ae850 con 0x7f81000abd50 2023-07-14T19:03:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.402+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 --> [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f811010ec30 con 0x7f81000abd50 2023-07-14T19:03:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.402+0000 7f810dffb700 1 -- 172.21.15.78:0/2601263423 <== osd.11 v2:172.21.15.102:6824/383662404 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f80fc00b790 con 0x7f81000abd50 2023-07-14T19:03:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] conn(0x7f81000abd50 msgr2=0x7f81000ae180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] conn(0x7f81000abd50 0x7f81000ae180 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f81000a11e0 msgr2=0x7f81000a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f81000a11e0 0x7f81000a3690 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8104011490 tx=0x7f8104009260 comp rx=0 tx=0).stop 2023-07-14T19:03:16.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8110103ee0 msgr2=0x7f811019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:16.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 --2- 172.21.15.78:0/2601263423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8110103ee0 0x7f811019a6c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f810800efc0 tx=0x7f810800c3a0 comp rx=0 tx=0).stop 2023-07-14T19:03:16.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f81154cf700 1 -- 172.21.15.78:0/2601263423 reap_dead start 2023-07-14T19:03:16.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 shutdown_connections 2023-07-14T19:03:16.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 >> 172.21.15.78:0/2601263423 conn(0x7f8110075ca0 msgr2=0x7f8110102790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:16.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 shutdown_connections 2023-07-14T19:03:16.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:16.403+0000 7f8116f32700 1 -- 172.21.15.78:0/2601263423 wait complete. 2023-07-14T19:03:16.880 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:16 smithi102 ceph-mon[129280]: pgmap v278: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:03:16.939 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:16 smithi078 ceph-mon[127425]: pgmap v278: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:03:16.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:16 smithi088 ceph-mon[128973]: pgmap v278: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:03:17.236 INFO:teuthology.orchestra.run.smithi078.stdout:326417514517 2023-07-14T19:03:17.236 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.11 2023-07-14T19:03:17.429 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.425+0000 7fb083233700 1 -- 172.21.15.78:0/463776932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb07c101830 msgr2=0x7fb07c109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:17.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.425+0000 7fb083233700 1 --2- 172.21.15.78:0/463776932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb07c101830 0x7fb07c109d30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb068009990 tx=0x7fb068009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:17.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.425+0000 7fb083233700 1 -- 172.21.15.78:0/463776932 shutdown_connections 2023-07-14T19:03:17.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.425+0000 7fb083233700 1 --2- 172.21.15.78:0/463776932 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb07c10a270 0x7fb07c10c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:17.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.425+0000 7fb083233700 1 --2- 172.21.15.78:0/463776932 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb07c100f20 0x7fb07c1012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:17.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.425+0000 7fb083233700 1 --2- 172.21.15.78:0/463776932 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb07c101830 0x7fb07c109d30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:17.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.425+0000 7fb083233700 1 -- 172.21.15.78:0/463776932 >> 172.21.15.78:0/463776932 conn(0x7fb07c075ca0 msgr2=0x7fb07c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:17.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 -- 172.21.15.78:0/463776932 shutdown_connections 2023-07-14T19:03:17.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 -- 172.21.15.78:0/463776932 wait complete. 2023-07-14T19:03:17.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 Processor -- start 2023-07-14T19:03:17.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 -- start start 2023-07-14T19:03:17.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb07c100f20 0x7fb07c195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:17.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb07c101830 0x7fb07c1962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:17.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb07c10a270 0x7fb07c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c19ad40 con 0x7fb07c100f20 2023-07-14T19:03:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c19aeb0 con 0x7fb07c101830 2023-07-14T19:03:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.426+0000 7fb083233700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07c19b070 con 0x7fb07c10a270 2023-07-14T19:03:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb07c100f20 0x7fb07c195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb07c100f20 0x7fb07c195d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35270/0 (socket says 172.21.15.78:35270) 2023-07-14T19:03:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 -- 172.21.15.78:0/1710264973 learned_addr learned my addr 172.21.15.78:0/1710264973 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb0817d0700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb07c10a270 0x7fb07c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb07bfff700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb07c101830 0x7fb07c1962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 -- 172.21.15.78:0/1710264973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb07c10a270 msgr2=0x7fb07c19a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb07c10a270 0x7fb07c19a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 -- 172.21.15.78:0/1710264973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb07c101830 msgr2=0x7fb07c1962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb07c101830 0x7fb07c1962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 -- 172.21.15.78:0/1710264973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb074009c40 con 0x7fb07c100f20 2023-07-14T19:03:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb080fcf700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb07c100f20 0x7fb07c195d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb07000b530 tx=0x7fb07000b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070011830 con 0x7fb07c100f20 2023-07-14T19:03:17.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0680095f0 con 0x7fb07c100f20 2023-07-14T19:03:17.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb070011e20 con 0x7fb07c100f20 2023-07-14T19:03:17.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.427+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb070010db0 con 0x7fb07c100f20 2023-07-14T19:03:17.435 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.428+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07c1a29e0 con 0x7fb07c100f20 2023-07-14T19:03:17.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.429+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb07c10fd40 con 0x7fb07c100f20 2023-07-14T19:03:17.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.430+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb070011990 con 0x7fb07c100f20 2023-07-14T19:03:17.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.431+0000 7fb079ffb700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb06c0a1260 0x7fb06c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:17.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.431+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fb0700cc4f0 con 0x7fb07c100f20 2023-07-14T19:03:17.436 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.431+0000 7fb07bfff700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb06c0a1260 0x7fb06c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:17.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.431+0000 7fb079ffb700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] conn(0x7fb06c0abdd0 0x7fb06c0ae200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:17.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.431+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 --> [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb06c0ae8d0 con 0x7fb06c0abdd0 2023-07-14T19:03:17.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.431+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_get_version_reply(handle=1 version=85) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb0700ceb70 con 0x7fb07c100f20 2023-07-14T19:03:17.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.431+0000 7fb07bfff700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb06c0a1260 0x7fb06c0a3710 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb068006010 tx=0x7fb068005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:17.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.431+0000 7fb0817d0700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] conn(0x7fb06c0abdd0 0x7fb06c0ae200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:17.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.432+0000 7fb0817d0700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] conn(0x7fb06c0abdd0 0x7fb06c0ae200 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:17.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.432+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 <== osd.5 v2:172.21.15.102:6808/3895937355 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25106 (crc 0 0 0) 0x7fb06c0ae8d0 con 0x7fb06c0abdd0 2023-07-14T19:03:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.448+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 --> [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb07c04e340 con 0x7fb06c0abdd0 2023-07-14T19:03:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.448+0000 7fb079ffb700 1 -- 172.21.15.78:0/1710264973 <== osd.5 v2:172.21.15.102:6808/3895937355 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb07c04e340 con 0x7fb06c0abdd0 2023-07-14T19:03:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 >> [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] conn(0x7fb06c0abdd0 msgr2=0x7fb06c0ae200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] conn(0x7fb06c0abdd0 0x7fb06c0ae200 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb06c0a1260 msgr2=0x7fb06c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb06c0a1260 0x7fb06c0a3710 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb068006010 tx=0x7fb068005960 comp rx=0 tx=0).stop 2023-07-14T19:03:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb07c100f20 msgr2=0x7fb07c195d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 --2- 172.21.15.78:0/1710264973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb07c100f20 0x7fb07c195d80 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb07000b530 tx=0x7fb07000b840 comp rx=0 tx=0).stop 2023-07-14T19:03:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb0817d0700 1 -- 172.21.15.78:0/1710264973 reap_dead start 2023-07-14T19:03:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 shutdown_connections 2023-07-14T19:03:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 >> 172.21.15.78:0/1710264973 conn(0x7fb07c075ca0 msgr2=0x7fb07c101c40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 shutdown_connections 2023-07-14T19:03:17.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:17.449+0000 7fb083233700 1 -- 172.21.15.78:0/1710264973 wait complete. 2023-07-14T19:03:17.793 INFO:teuthology.orchestra.run.smithi078.stdout:193273528362 2023-07-14T19:03:17.794 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.6 2023-07-14T19:03:17.960 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:17.983 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:18.511 INFO:teuthology.orchestra.run.smithi078.stdout:163208757294 2023-07-14T19:03:18.511 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph osd last-stat-seq osd.5 2023-07-14T19:03:18.837 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:19.265 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:19.265 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:19.432 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:19 smithi078 ceph-mon[127425]: pgmap v279: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:19.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:19 smithi088 ceph-mon[128973]: pgmap v279: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:19.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:19 smithi102 ceph-mon[129280]: pgmap v279: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:19.843 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:20.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 -- 172.21.15.78:0/2069034334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40fff20 msgr2=0x7f8af4103820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:20.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 --2- 172.21.15.78:0/2069034334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40fff20 0x7f8af4103820 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0099c0 tx=0x7f8adc009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 -- 172.21.15.78:0/2069034334 shutdown_connections 2023-07-14T19:03:20.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 --2- 172.21.15.78:0/2069034334 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af40ff580 0x7f8af40ff950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 --2- 172.21.15.78:0/2069034334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8af4103f50 0x7f8af410bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 --2- 172.21.15.78:0/2069034334 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40fff20 0x7f8af4103820 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 -- 172.21.15.78:0/2069034334 >> 172.21.15.78:0/2069034334 conn(0x7f8af4075280 msgr2=0x7f8af4075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:20.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 -- 172.21.15.78:0/2069034334 shutdown_connections 2023-07-14T19:03:20.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.945+0000 7f8af9c80700 1 -- 172.21.15.78:0/2069034334 wait complete. 2023-07-14T19:03:20.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af9c80700 1 Processor -- start 2023-07-14T19:03:20.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af9c80700 1 -- start start 2023-07-14T19:03:20.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af9c80700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40ff580 0x7f8af4196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:20.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af9c80700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af40fff20 0x7f8af41970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:20.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af9c80700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8af4103f50 0x7f8af41a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af9c80700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af41979e0 con 0x7f8af40ff580 2023-07-14T19:03:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af9c80700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af41a2120 con 0x7f8af40fff20 2023-07-14T19:03:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af9c80700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af41a22e0 con 0x7f8af4103f50 2023-07-14T19:03:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40ff580 0x7f8af4196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af37fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40ff580 0x7f8af4196b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35286/0 (socket says 172.21.15.78:35286) 2023-07-14T19:03:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8af37fe700 1 -- 172.21.15.78:0/1088208917 learned_addr learned my addr 172.21.15.78:0/1088208917 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.946+0000 7f8aebfff700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af40fff20 0x7f8af41970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:20.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af37fe700 1 -- 172.21.15.78:0/1088208917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8af4103f50 msgr2=0x7f8af41a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:20.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af3fff700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8af4103f50 0x7f8af41a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:20.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af37fe700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8af4103f50 0x7f8af41a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af37fe700 1 -- 172.21.15.78:0/1088208917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af40fff20 msgr2=0x7f8af41970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:20.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af37fe700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af40fff20 0x7f8af41970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af37fe700 1 -- 172.21.15.78:0/1088208917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc0095f0 con 0x7f8af40ff580 2023-07-14T19:03:20.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af3fff700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8af4103f50 0x7f8af41a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:20.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af37fe700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40ff580 0x7f8af4196b60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400d950 tx=0x7f8ae400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:20.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af17fa700 1 -- 172.21.15.78:0/1088208917 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4009940 con 0x7f8af40ff580 2023-07-14T19:03:20.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af17fa700 1 -- 172.21.15.78:0/1088208917 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8ae4010460 con 0x7f8af40ff580 2023-07-14T19:03:20.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af41a25c0 con 0x7f8af40ff580 2023-07-14T19:03:20.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af17fa700 1 -- 172.21.15.78:0/1088208917 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae400f620 con 0x7f8af40ff580 2023-07-14T19:03:20.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.947+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af41a2a90 con 0x7f8af40ff580 2023-07-14T19:03:20.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.948+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af404e340 con 0x7f8af40ff580 2023-07-14T19:03:20.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.949+0000 7f8af17fa700 1 -- 172.21.15.78:0/1088208917 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8ae40105d0 con 0x7f8af40ff580 2023-07-14T19:03:20.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.950+0000 7f8af17fa700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ae00a11e0 0x7f8ae00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:20.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.950+0000 7f8aebfff700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ae00a11e0 0x7f8ae00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:20.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.950+0000 7f8af17fa700 1 -- 172.21.15.78:0/1088208917 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f8ae40cca80 con 0x7f8af40ff580 2023-07-14T19:03:20.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.950+0000 7f8aebfff700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ae00a11e0 0x7f8ae00a3690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8af4103860 tx=0x7f8adc009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:20.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:19.953+0000 7f8af17fa700 1 -- 172.21.15.78:0/1088208917 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8ae4095e80 con 0x7f8af40ff580 2023-07-14T19:03:20.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.148+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f8af41a2dc0 con 0x7f8af40ff580 2023-07-14T19:03:20.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.148+0000 7f8af17fa700 1 -- 172.21.15.78:0/1088208917 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8ae4016090 con 0x7f8af40ff580 2023-07-14T19:03:20.152 INFO:teuthology.orchestra.run.smithi078.stdout:55834574910 2023-07-14T19:03:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ae00a11e0 msgr2=0x7f8ae00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ae00a11e0 0x7f8ae00a3690 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8af4103860 tx=0x7f8adc009f10 comp rx=0 tx=0).stop 2023-07-14T19:03:20.154 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40ff580 msgr2=0x7f8af4196b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40ff580 0x7f8af4196b60 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400d950 tx=0x7f8ae400dc60 comp rx=0 tx=0).stop 2023-07-14T19:03:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 shutdown_connections 2023-07-14T19:03:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ae00a11e0 0x7f8ae00a3690 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8af4103f50 0x7f8af41a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8af40ff580 0x7f8af4196b60 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.150+0000 7f8af9c80700 1 --2- 172.21.15.78:0/1088208917 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8af40fff20 0x7f8af41970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.151+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 >> 172.21.15.78:0/1088208917 conn(0x7f8af4075280 msgr2=0x7f8af40fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:20.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.151+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 shutdown_connections 2023-07-14T19:03:20.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:20.151+0000 7f8af9c80700 1 -- 172.21.15.78:0/1088208917 wait complete. 2023-07-14T19:03:20.414 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:20 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1088208917' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:03:20.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:20 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1088208917' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:03:20.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:20 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1088208917' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-07-14T19:03:21.161 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:21.189 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574908 got 55834574910 for osd.1 2023-07-14T19:03:21.189 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:21.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:21 smithi088 ceph-mon[128973]: pgmap v280: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:21.516 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:21 smithi078 ceph-mon[127425]: pgmap v280: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:21.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:21 smithi102 ceph-mon[129280]: pgmap v280: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:22.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 -- 172.21.15.78:0/2258444104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d80fee40 msgr2=0x7f26d80ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 --2- 172.21.15.78:0/2258444104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d80fee40 0x7f26d80ff210 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0099c0 tx=0x7f26cc009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 -- 172.21.15.78:0/2258444104 shutdown_connections 2023-07-14T19:03:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 --2- 172.21.15.78:0/2258444104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26d8103ee0 0x7f26d810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 --2- 172.21.15.78:0/2258444104 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d80fee40 0x7f26d80ff210 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 --2- 172.21.15.78:0/2258444104 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26d80ff7e0 0x7f26d81037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 -- 172.21.15.78:0/2258444104 >> 172.21.15.78:0/2258444104 conn(0x7f26d8075ca0 msgr2=0x7f26d80760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:22.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 -- 172.21.15.78:0/2258444104 shutdown_connections 2023-07-14T19:03:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.098+0000 7f26dedeb700 1 -- 172.21.15.78:0/2258444104 wait complete. 2023-07-14T19:03:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dedeb700 1 Processor -- start 2023-07-14T19:03:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dedeb700 1 -- start start 2023-07-14T19:03:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dedeb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26d80fee40 0x7f26d8100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dedeb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26d80ff7e0 0x7f26d8100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dedeb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d8103ee0 0x7f26d8101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dedeb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d81a4160 con 0x7f26d8103ee0 2023-07-14T19:03:22.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dedeb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d81a42d0 con 0x7f26d80fee40 2023-07-14T19:03:22.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dedeb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26d81a4490 con 0x7f26d80ff7e0 2023-07-14T19:03:22.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dd388700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d8103ee0 0x7f26d8101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dd388700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d8103ee0 0x7f26d8101310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35302/0 (socket says 172.21.15.78:35302) 2023-07-14T19:03:22.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dd388700 1 -- 172.21.15.78:0/706009308 learned_addr learned my addr 172.21.15.78:0/706009308 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:22.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dcb87700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26d80fee40 0x7f26d8100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26d7fff700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26d80ff7e0 0x7f26d8100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dd388700 1 -- 172.21.15.78:0/706009308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26d80ff7e0 msgr2=0x7f26d8100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dd388700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26d80ff7e0 0x7f26d8100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dd388700 1 -- 172.21.15.78:0/706009308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26d80fee40 msgr2=0x7f26d8100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dd388700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26d80fee40 0x7f26d8100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.099+0000 7f26dd388700 1 -- 172.21.15.78:0/706009308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26cc0095f0 con 0x7f26d8103ee0 2023-07-14T19:03:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.100+0000 7f26dd388700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d8103ee0 0x7f26d8101310 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f26d0007f10 tx=0x7f26d000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.100+0000 7f26d5ffb700 1 -- 172.21.15.78:0/706009308 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0009ea0 con 0x7f26d8103ee0 2023-07-14T19:03:22.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.100+0000 7f26d5ffb700 1 -- 172.21.15.78:0/706009308 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f26d000f040 con 0x7f26d8103ee0 2023-07-14T19:03:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.100+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26d81a4770 con 0x7f26d8103ee0 2023-07-14T19:03:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.100+0000 7f26d5ffb700 1 -- 172.21.15.78:0/706009308 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26d0004640 con 0x7f26d8103ee0 2023-07-14T19:03:22.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.100+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26d81a4c20 con 0x7f26d8103ee0 2023-07-14T19:03:22.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.101+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26d804e340 con 0x7f26d8103ee0 2023-07-14T19:03:22.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.102+0000 7f26d5ffb700 1 -- 172.21.15.78:0/706009308 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f26d00047a0 con 0x7f26d8103ee0 2023-07-14T19:03:22.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.102+0000 7f26d5ffb700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f26c80a1260 0x7f26c80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.102+0000 7f26dcb87700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f26c80a1260 0x7f26c80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.103+0000 7f26d5ffb700 1 -- 172.21.15.78:0/706009308 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f26d00cc4c0 con 0x7f26d8103ee0 2023-07-14T19:03:22.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.103+0000 7f26dcb87700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f26c80a1260 0x7f26c80a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00b580 tx=0x7f26cc009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:22.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.107+0000 7f26d5ffb700 1 -- 172.21.15.78:0/706009308 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f26d004e780 con 0x7f26d8103ee0 2023-07-14T19:03:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.310+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f26d81a4e40 con 0x7f26d8103ee0 2023-07-14T19:03:22.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.310+0000 7f26d5ffb700 1 -- 172.21.15.78:0/706009308 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f26d0095860 con 0x7f26d8103ee0 2023-07-14T19:03:22.316 INFO:teuthology.orchestra.run.smithi078.stdout:214748364839 2023-07-14T19:03:22.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f26c80a1260 msgr2=0x7f26c80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f26c80a1260 0x7f26c80a3710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00b580 tx=0x7f26cc009f10 comp rx=0 tx=0).stop 2023-07-14T19:03:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d8103ee0 msgr2=0x7f26d8101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d8103ee0 0x7f26d8101310 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f26d0007f10 tx=0x7f26d000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 shutdown_connections 2023-07-14T19:03:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f26c80a1260 0x7f26c80a3710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f26d80fee40 0x7f26d8100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f26d8103ee0 0x7f26d8101310 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 --2- 172.21.15.78:0/706009308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26d80ff7e0 0x7f26d8100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 >> 172.21.15.78:0/706009308 conn(0x7f26d8075ca0 msgr2=0x7f26d80681a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 shutdown_connections 2023-07-14T19:03:22.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.313+0000 7f26dedeb700 1 -- 172.21.15.78:0/706009308 wait complete. 2023-07-14T19:03:22.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 -- 172.21.15.78:0/2033311271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c0fff40 msgr2=0x7fa12c100310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.445 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 --2- 172.21.15.78:0/2033311271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c0fff40 0x7fa12c100310 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa11c00b0d0 tx=0x7fa11c00b3e0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 -- 172.21.15.78:0/2033311271 shutdown_connections 2023-07-14T19:03:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 --2- 172.21.15.78:0/2033311271 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c104fe0 0x7fa12c10cb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 --2- 172.21.15.78:0/2033311271 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c0fff40 0x7fa12c100310 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 --2- 172.21.15.78:0/2033311271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa12c1008e0 0x7fa12c1048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 -- 172.21.15.78:0/2033311271 >> 172.21.15.78:0/2033311271 conn(0x7fa12c0fb7c0 msgr2=0x7fa12c0fdbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:22.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 -- 172.21.15.78:0/2033311271 shutdown_connections 2023-07-14T19:03:22.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.440+0000 7fa131a0f700 1 -- 172.21.15.78:0/2033311271 wait complete. 2023-07-14T19:03:22.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.441+0000 7fa131a0f700 1 Processor -- start 2023-07-14T19:03:22.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.441+0000 7fa131a0f700 1 -- start start 2023-07-14T19:03:22.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa131a0f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa12c0fff40 0x7fa12c195e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa131a0f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c1008e0 0x7fa12c1963b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa131a0f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c104fe0 0x7fa12c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa131a0f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c19ae30 con 0x7fa12c1008e0 2023-07-14T19:03:22.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa131a0f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c19afa0 con 0x7fa12c104fe0 2023-07-14T19:03:22.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa131a0f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12c19b160 con 0x7fa12c0fff40 2023-07-14T19:03:22.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c1008e0 0x7fa12c1963b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c1008e0 0x7fa12c1963b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35312/0 (socket says 172.21.15.78:35312) 2023-07-14T19:03:22.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 -- 172.21.15.78:0/3297540344 learned_addr learned my addr 172.21.15.78:0/3297540344 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:22.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12affd700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa12c0fff40 0x7fa12c195e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12b7fe700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c104fe0 0x7fa12c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 -- 172.21.15.78:0/3297540344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa12c0fff40 msgr2=0x7fa12c195e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa12c0fff40 0x7fa12c195e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.450 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 -- 172.21.15.78:0/3297540344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c104fe0 msgr2=0x7fa12c19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c104fe0 0x7fa12c19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 -- 172.21.15.78:0/3297540344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa11c009d00 con 0x7fa12c1008e0 2023-07-14T19:03:22.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12b7fe700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c104fe0 0x7fa12c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:22.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa12a7fc700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c1008e0 0x7fa12c1963b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa11400b500 tx=0x7fa11400b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:22.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa130a0d700 1 -- 172.21.15.78:0/3297540344 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114011830 con 0x7fa12c1008e0 2023-07-14T19:03:22.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa130a0d700 1 -- 172.21.15.78:0/3297540344 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa114011e20 con 0x7fa12c1008e0 2023-07-14T19:03:22.451 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa12c19b440 con 0x7fa12c1008e0 2023-07-14T19:03:22.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.442+0000 7fa130a0d700 1 -- 172.21.15.78:0/3297540344 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa114010db0 con 0x7fa12c1008e0 2023-07-14T19:03:22.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.443+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa12c1a2ca0 con 0x7fa12c1008e0 2023-07-14T19:03:22.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.443+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12c04e340 con 0x7fa12c1008e0 2023-07-14T19:03:22.452 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.445+0000 7fa130a0d700 1 -- 172.21.15.78:0/3297540344 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa114011990 con 0x7fa12c1008e0 2023-07-14T19:03:22.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.445+0000 7fa130a0d700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa1180a1330 0x7fa1180a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.445+0000 7fa12affd700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa1180a1330 0x7fa1180a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.445+0000 7fa130a0d700 1 -- 172.21.15.78:0/3297540344 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fa1140cc4f0 con 0x7fa12c1008e0 2023-07-14T19:03:22.453 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.446+0000 7fa12affd700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa1180a1330 0x7fa1180a37e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009fd0 tx=0x7fa11c00b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:22.454 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.449+0000 7fa130a0d700 1 -- 172.21.15.78:0/3297540344 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa114095a00 con 0x7fa12c1008e0 2023-07-14T19:03:22.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.661+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7fa12c1971d0 con 0x7fa12c1008e0 2023-07-14T19:03:22.666 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.661+0000 7fa130a0d700 1 -- 172.21.15.78:0/3297540344 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fa114099020 con 0x7fa12c1008e0 2023-07-14T19:03:22.666 INFO:teuthology.orchestra.run.smithi078.stdout:274877906973 2023-07-14T19:03:22.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.663+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa1180a1330 msgr2=0x7fa1180a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.663+0000 7fa131a0f700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa1180a1330 0x7fa1180a37e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa11c009fd0 tx=0x7fa11c00b040 comp rx=0 tx=0).stop 2023-07-14T19:03:22.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.663+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c1008e0 msgr2=0x7fa12c1963b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.663+0000 7fa131a0f700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c1008e0 0x7fa12c1963b0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa11400b500 tx=0x7fa11400b810 comp rx=0 tx=0).stop 2023-07-14T19:03:22.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.664+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 shutdown_connections 2023-07-14T19:03:22.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.664+0000 7fa131a0f700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa1180a1330 0x7fa1180a37e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.664+0000 7fa131a0f700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa12c104fe0 0x7fa12c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.664+0000 7fa131a0f700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa12c0fff40 0x7fa12c195e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.664+0000 7fa131a0f700 1 --2- 172.21.15.78:0/3297540344 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa12c1008e0 0x7fa12c1963b0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.664+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 >> 172.21.15.78:0/3297540344 conn(0x7fa12c0fb7c0 msgr2=0x7fa12c103930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:22.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.664+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 shutdown_connections 2023-07-14T19:03:22.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.664+0000 7fa131a0f700 1 -- 172.21.15.78:0/3297540344 wait complete. 2023-07-14T19:03:22.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 -- 172.21.15.78:0/697742423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 msgr2=0x7efc581012f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 --2- 172.21.15.78:0/697742423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 0x7efc581012f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7efc48009990 tx=0x7efc48009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 -- 172.21.15.78:0/697742423 shutdown_connections 2023-07-14T19:03:22.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 --2- 172.21.15.78:0/697742423 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc5810a270 0x7efc5810c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 --2- 172.21.15.78:0/697742423 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 0x7efc581012f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 --2- 172.21.15.78:0/697742423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc58101830 0x7efc58109d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 -- 172.21.15.78:0/697742423 >> 172.21.15.78:0/697742423 conn(0x7efc58075ca0 msgr2=0x7efc580760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:22.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 -- 172.21.15.78:0/697742423 shutdown_connections 2023-07-14T19:03:22.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.676+0000 7efc5c852700 1 -- 172.21.15.78:0/697742423 wait complete. 2023-07-14T19:03:22.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.677+0000 7efc5c852700 1 Processor -- start 2023-07-14T19:03:22.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.677+0000 7efc5c852700 1 -- start start 2023-07-14T19:03:22.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5c852700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 0x7efc580feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5c852700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc58101830 0x7efc580ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5c852700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc5810a270 0x7efc580ff600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.684 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5c852700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc581a8610 con 0x7efc58100f20 2023-07-14T19:03:22.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5c852700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc58199dd0 con 0x7efc5810a270 2023-07-14T19:03:22.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5c852700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc58199f90 con 0x7efc58101830 2023-07-14T19:03:22.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 0x7efc580feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 0x7efc580feb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35322/0 (socket says 172.21.15.78:35322) 2023-07-14T19:03:22.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc55d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc58101830 0x7efc580ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 -- 172.21.15.78:0/1997975360 learned_addr learned my addr 172.21.15.78:0/1997975360 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:22.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc56d9d700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc5810a270 0x7efc580ff600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 -- 172.21.15.78:0/1997975360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc58101830 msgr2=0x7efc580ff0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc58101830 0x7efc580ff0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 -- 172.21.15.78:0/1997975360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc5810a270 msgr2=0x7efc580ff600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc5810a270 0x7efc580ff600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 -- 172.21.15.78:0/1997975360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc480095f0 con 0x7efc58100f20 2023-07-14T19:03:22.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc56d9d700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc5810a270 0x7efc580ff600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:22.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.678+0000 7efc5659c700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 0x7efc580feb80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efc48005910 tx=0x7efc480047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:22.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.679+0000 7efc477fe700 1 -- 172.21.15.78:0/1997975360 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4801d070 con 0x7efc58100f20 2023-07-14T19:03:22.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.679+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5819a210 con 0x7efc58100f20 2023-07-14T19:03:22.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.679+0000 7efc477fe700 1 -- 172.21.15.78:0/1997975360 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7efc48021410 con 0x7efc58100f20 2023-07-14T19:03:22.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.679+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5819a700 con 0x7efc58100f20 2023-07-14T19:03:22.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.679+0000 7efc477fe700 1 -- 172.21.15.78:0/1997975360 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48017650 con 0x7efc58100f20 2023-07-14T19:03:22.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.680+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc38005320 con 0x7efc58100f20 2023-07-14T19:03:22.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.681+0000 7efc477fe700 1 -- 172.21.15.78:0/1997975360 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7efc4800fb00 con 0x7efc58100f20 2023-07-14T19:03:22.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.682+0000 7efc477fe700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efc400a11e0 0x7efc400a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:22.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.682+0000 7efc55d9b700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efc400a11e0 0x7efc400a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:22.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.682+0000 7efc477fe700 1 -- 172.21.15.78:0/1997975360 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7efc480ce570 con 0x7efc58100f20 2023-07-14T19:03:22.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.683+0000 7efc55d9b700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efc400a11e0 0x7efc400a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efc58100370 tx=0x7efc4c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:22.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.687+0000 7efc477fe700 1 -- 172.21.15.78:0/1997975360 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7efc48097a00 con 0x7efc58100f20 2023-07-14T19:03:22.903 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.899+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7efc38005190 con 0x7efc58100f20 2023-07-14T19:03:22.904 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.899+0000 7efc477fe700 1 -- 172.21.15.78:0/1997975360 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7efc4809b020 con 0x7efc58100f20 2023-07-14T19:03:22.904 INFO:teuthology.orchestra.run.smithi078.stdout:77309411388 2023-07-14T19:03:22.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.901+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efc400a11e0 msgr2=0x7efc400a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.901+0000 7efc5c852700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efc400a11e0 0x7efc400a3690 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efc58100370 tx=0x7efc4c00b410 comp rx=0 tx=0).stop 2023-07-14T19:03:22.906 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.901+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 msgr2=0x7efc580feb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.901+0000 7efc5c852700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 0x7efc580feb80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7efc48005910 tx=0x7efc480047b0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.902+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 shutdown_connections 2023-07-14T19:03:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.902+0000 7efc5c852700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7efc400a11e0 0x7efc400a3690 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.902+0000 7efc5c852700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7efc5810a270 0x7efc580ff600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.902+0000 7efc5c852700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7efc58100f20 0x7efc580feb80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.902+0000 7efc5c852700 1 --2- 172.21.15.78:0/1997975360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc58101830 0x7efc580ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.902+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 >> 172.21.15.78:0/1997975360 conn(0x7efc58075ca0 msgr2=0x7efc58101c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:22.907 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.902+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 shutdown_connections 2023-07-14T19:03:22.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:22.902+0000 7efc5c852700 1 -- 172.21.15.78:0/1997975360 wait complete. 2023-07-14T19:03:23.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:23 smithi088 ceph-mon[128973]: pgmap v281: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:23.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:23 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/706009308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:03:23.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:23 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3297540344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:03:23.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:23 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1997975360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:03:23.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:23 smithi078 ceph-mon[127425]: pgmap v281: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:23.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:23 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/706009308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:03:23.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:23 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3297540344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:03:23.531 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:23 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1997975360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:03:23.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:23 smithi102 ceph-mon[129280]: pgmap v281: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:03:23.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:23 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/706009308' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-07-14T19:03:23.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:23 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3297540344' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2023-07-14T19:03:23.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:23 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1997975360' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-07-14T19:03:23.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4121584295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 msgr2=0x7f4d040ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:23.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4121584295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 0x7f4d040ff310 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f4cf80099c0 tx=0x7f4cf8009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4121584295 shutdown_connections 2023-07-14T19:03:23.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4121584295 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d04101150 0x7f4d0410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4121584295 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 0x7f4d040ff310 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4121584295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d041007a0 0x7f4d04100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4121584295 >> 172.21.15.78:0/4121584295 conn(0x7f4d04075c90 msgr2=0x7f4d04076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:23.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4121584295 shutdown_connections 2023-07-14T19:03:23.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.642+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4121584295 wait complete. 2023-07-14T19:03:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0c19a700 1 Processor -- start 2023-07-14T19:03:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0c19a700 1 -- start start 2023-07-14T19:03:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0c19a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 0x7f4d0419ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:23.650 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0c19a700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d041007a0 0x7f4d0419bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0c19a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d04101150 0x7f4d04195ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0c19a700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d04196120 con 0x7f4d040fef40 2023-07-14T19:03:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0c19a700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d04196290 con 0x7f4d041007a0 2023-07-14T19:03:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0c19a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d04196450 con 0x7f4d04101150 2023-07-14T19:03:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d09f36700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 0x7f4d0419ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d09f36700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 0x7f4d0419ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35334/0 (socket says 172.21.15.78:35334) 2023-07-14T19:03:23.651 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d09f36700 1 -- 172.21.15.78:0/4006786450 learned_addr learned my addr 172.21.15.78:0/4006786450 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.643+0000 7f4d0a737700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d04101150 0x7f4d04195ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d09735700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d041007a0 0x7f4d0419bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d09f36700 1 -- 172.21.15.78:0/4006786450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d04101150 msgr2=0x7f4d04195ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d09f36700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d04101150 0x7f4d04195ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d09f36700 1 -- 172.21.15.78:0/4006786450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d041007a0 msgr2=0x7f4d0419bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d09f36700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d041007a0 0x7f4d0419bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.652 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d09f36700 1 -- 172.21.15.78:0/4006786450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf80095f0 con 0x7f4d040fef40 2023-07-14T19:03:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d0a737700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d04101150 0x7f4d04195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d09f36700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 0x7f4d0419ba30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009f50 tx=0x7f4cf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4cf6ffd700 1 -- 172.21.15.78:0/4006786450 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf801d070 con 0x7f4d040fef40 2023-07-14T19:03:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4cf6ffd700 1 -- 172.21.15.78:0/4006786450 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4cf80040b0 con 0x7f4d040fef40 2023-07-14T19:03:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d041966d0 con 0x7f4d040fef40 2023-07-14T19:03:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4cf6ffd700 1 -- 172.21.15.78:0/4006786450 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cf8022cd0 con 0x7f4d040fef40 2023-07-14T19:03:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.644+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d041a2a20 con 0x7f4d040fef40 2023-07-14T19:03:23.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.645+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d0404e340 con 0x7f4d040fef40 2023-07-14T19:03:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.646+0000 7f4cf6ffd700 1 -- 172.21.15.78:0/4006786450 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4cf8022470 con 0x7f4d040fef40 2023-07-14T19:03:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.646+0000 7f4cf6ffd700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4cf00a1260 0x7f4cf00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:23.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.647+0000 7f4d09735700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4cf00a1260 0x7f4cf00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.647+0000 7f4cf6ffd700 1 -- 172.21.15.78:0/4006786450 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f4cf80ce830 con 0x7f4d040fef40 2023-07-14T19:03:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.647+0000 7f4d09735700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4cf00a1260 0x7f4cf00a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4d00005fd0 tx=0x7f4d00005f60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:23.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.650+0000 7f4cf6ffd700 1 -- 172.21.15.78:0/4006786450 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4cf8097c60 con 0x7f4d040fef40 2023-07-14T19:03:23.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.845+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f4d04196fc0 con 0x7f4d040fef40 2023-07-14T19:03:23.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.846+0000 7f4cf6ffd700 1 -- 172.21.15.78:0/4006786450 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f4cf809b280 con 0x7f4d040fef40 2023-07-14T19:03:23.850 INFO:teuthology.orchestra.run.smithi078.stdout:244813135906 2023-07-14T19:03:23.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4cf00a1260 msgr2=0x7f4cf00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:23.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4cf00a1260 0x7f4cf00a3710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f4d00005fd0 tx=0x7f4d00005f60 comp rx=0 tx=0).stop 2023-07-14T19:03:23.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 msgr2=0x7f4d0419ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:23.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 0x7f4d0419ba30 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f4cf8009f50 tx=0x7f4cf80057f0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 shutdown_connections 2023-07-14T19:03:23.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4cf00a1260 0x7f4cf00a3710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d04101150 0x7f4d04195ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4d040fef40 0x7f4d0419ba30 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 --2- 172.21.15.78:0/4006786450 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4d041007a0 0x7f4d0419bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:23.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 >> 172.21.15.78:0/4006786450 conn(0x7f4d04075c90 msgr2=0x7f4d040fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:23.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 shutdown_connections 2023-07-14T19:03:23.854 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:23.848+0000 7f4d0c19a700 1 -- 172.21.15.78:0/4006786450 wait complete. 2023-07-14T19:03:23.856 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411386 got 77309411388 for osd.2 2023-07-14T19:03:23.856 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:23.995 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:24.161 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:24.193 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:24 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/4006786450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:03:24.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:24 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/4006786450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:03:24.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:24 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/4006786450' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2023-07-14T19:03:24.752 INFO:tasks.cephadm.ceph_manager.ceph:need seq 244813135904 got 244813135906 for osd.8 2023-07-14T19:03:24.752 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:24.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 -- 172.21.15.78:0/4031973093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f41018c0 msgr2=0x7f35f4109d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:24.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 --2- 172.21.15.78:0/4031973093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f41018c0 0x7f35f4109d30 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f35e4009990 tx=0x7f35e4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 -- 172.21.15.78:0/4031973093 shutdown_connections 2023-07-14T19:03:24.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 --2- 172.21.15.78:0/4031973093 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35f410a3a0 0x7f35f410c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 --2- 172.21.15.78:0/4031973093 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f4100f20 0x7f35f41012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 --2- 172.21.15.78:0/4031973093 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f41018c0 0x7f35f4109d30 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 -- 172.21.15.78:0/4031973093 >> 172.21.15.78:0/4031973093 conn(0x7f35f4075c90 msgr2=0x7f35f4076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:24.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 -- 172.21.15.78:0/4031973093 shutdown_connections 2023-07-14T19:03:24.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.772+0000 7f35fc108700 1 -- 172.21.15.78:0/4031973093 wait complete. 2023-07-14T19:03:24.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fc108700 1 Processor -- start 2023-07-14T19:03:24.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fc108700 1 -- start start 2023-07-14T19:03:24.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fc108700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f4100f20 0x7f35f40feb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:24.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fc108700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f41018c0 0x7f35f40ff0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:24.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fc108700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35f410a3a0 0x7f35f40ff640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:24.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fc108700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f41a41a0 con 0x7f35f4100f20 2023-07-14T19:03:24.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fc108700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f41a4310 con 0x7f35f410a3a0 2023-07-14T19:03:24.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fc108700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f41a44d0 con 0x7f35f41018c0 2023-07-14T19:03:24.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f9ea4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f4100f20 0x7f35f40feb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:24.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f9ea4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f4100f20 0x7f35f40feb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35348/0 (socket says 172.21.15.78:35348) 2023-07-14T19:03:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f9ea4700 1 -- 172.21.15.78:0/1470172467 learned_addr learned my addr 172.21.15.78:0/1470172467 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f96a3700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f41018c0 0x7f35f40ff0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fa6a5700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35f410a3a0 0x7f35f40ff640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f9ea4700 1 -- 172.21.15.78:0/1470172467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f41018c0 msgr2=0x7f35f40ff0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f9ea4700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f41018c0 0x7f35f40ff0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f9ea4700 1 -- 172.21.15.78:0/1470172467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35f410a3a0 msgr2=0x7f35f40ff640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f9ea4700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35f410a3a0 0x7f35f40ff640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35f9ea4700 1 -- 172.21.15.78:0/1470172467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e40095f0 con 0x7f35f4100f20 2023-07-14T19:03:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.773+0000 7f35fa6a5700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35f410a3a0 0x7f35f40ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.774+0000 7f35f9ea4700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f4100f20 0x7f35f40feb80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f35f000d950 tx=0x7f35f000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.774+0000 7f35eaffd700 1 -- 172.21.15.78:0/1470172467 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f0009940 con 0x7f35f4100f20 2023-07-14T19:03:24.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.774+0000 7f35eaffd700 1 -- 172.21.15.78:0/1470172467 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f35f0010460 con 0x7f35f4100f20 2023-07-14T19:03:24.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.774+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f41a47b0 con 0x7f35f4100f20 2023-07-14T19:03:24.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.774+0000 7f35eaffd700 1 -- 172.21.15.78:0/1470172467 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35f000f620 con 0x7f35f4100f20 2023-07-14T19:03:24.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.774+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f41a4c80 con 0x7f35f4100f20 2023-07-14T19:03:24.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.775+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f404e340 con 0x7f35f4100f20 2023-07-14T19:03:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.776+0000 7f35eaffd700 1 -- 172.21.15.78:0/1470172467 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f35f00105d0 con 0x7f35f4100f20 2023-07-14T19:03:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.776+0000 7f35eaffd700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f35e00a11e0 0x7f35e00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.776+0000 7f35f96a3700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f35e00a11e0 0x7f35e00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.776+0000 7f35eaffd700 1 -- 172.21.15.78:0/1470172467 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f35f00ccae0 con 0x7f35f4100f20 2023-07-14T19:03:24.785 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.777+0000 7f35f96a3700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f35e00a11e0 0x7f35e00a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f35e400b5c0 tx=0x7f35e4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:24.786 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.781+0000 7f35eaffd700 1 -- 172.21.15.78:0/1470172467 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f35f0095ee0 con 0x7f35f4100f20 2023-07-14T19:03:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.989+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f35f41000b0 con 0x7f35f4100f20 2023-07-14T19:03:24.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.990+0000 7f35eaffd700 1 -- 172.21.15.78:0/1470172467 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f35f0016090 con 0x7f35f4100f20 2023-07-14T19:03:24.995 INFO:teuthology.orchestra.run.smithi078.stdout:296352743450 2023-07-14T19:03:24.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f35e00a11e0 msgr2=0x7f35e00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:24.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f35e00a11e0 0x7f35e00a3690 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f35e400b5c0 tx=0x7f35e4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f4100f20 msgr2=0x7f35f40feb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:24.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f4100f20 0x7f35f40feb80 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f35f000d950 tx=0x7f35f000dc60 comp rx=0 tx=0).stop 2023-07-14T19:03:24.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 shutdown_connections 2023-07-14T19:03:24.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f35e00a11e0 0x7f35e00a3690 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35f410a3a0 0x7f35f40ff640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35f4100f20 0x7f35f40feb80 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 --2- 172.21.15.78:0/1470172467 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35f41018c0 0x7f35f40ff0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:24.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 >> 172.21.15.78:0/1470172467 conn(0x7f35f4075c90 msgr2=0x7f35f40fd470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:24.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 shutdown_connections 2023-07-14T19:03:24.998 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:24.992+0000 7f35fc108700 1 -- 172.21.15.78:0/1470172467 wait complete. 2023-07-14T19:03:25.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 -- 172.21.15.78:0/2877741130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c103ee0 msgr2=0x7f2e5c10ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:25.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 --2- 172.21.15.78:0/2877741130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c103ee0 0x7f2e5c10ba60 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f2e4c0099c0 tx=0x7f2e4c009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 -- 172.21.15.78:0/2877741130 shutdown_connections 2023-07-14T19:03:25.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 --2- 172.21.15.78:0/2877741130 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c103ee0 0x7f2e5c10ba60 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 --2- 172.21.15.78:0/2877741130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e5c0fee40 0x7f2e5c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 --2- 172.21.15.78:0/2877741130 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e5c0ff7e0 0x7f2e5c1037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 -- 172.21.15.78:0/2877741130 >> 172.21.15.78:0/2877741130 conn(0x7f2e5c075ca0 msgr2=0x7f2e5c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:25.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 -- 172.21.15.78:0/2877741130 shutdown_connections 2023-07-14T19:03:25.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 -- 172.21.15.78:0/2877741130 wait complete. 2023-07-14T19:03:25.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.052+0000 7f2e643c5700 1 Processor -- start 2023-07-14T19:03:25.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e643c5700 1 -- start start 2023-07-14T19:03:25.061 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e643c5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e5c0fee40 0x7f2e5c100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:25.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e643c5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c0ff7e0 0x7f2e5c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:25.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e643c5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e5c103ee0 0x7f2e5c101310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:25.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e643c5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c1a40a0 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e643c5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c1a4210 con 0x7f2e5c103ee0 2023-07-14T19:03:25.062 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e643c5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e5c1a43d0 con 0x7f2e5c0fee40 2023-07-14T19:03:25.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c0ff7e0 0x7f2e5c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:25.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c0ff7e0 0x7f2e5c100d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35364/0 (socket says 172.21.15.78:35364) 2023-07-14T19:03:25.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 -- 172.21.15.78:0/844968610 learned_addr learned my addr 172.21.15.78:0/844968610 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:25.063 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e62161700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e5c0fee40 0x7f2e5c100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:25.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e62962700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e5c103ee0 0x7f2e5c101310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:25.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 -- 172.21.15.78:0/844968610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e5c0fee40 msgr2=0x7f2e5c100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:25.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e5c0fee40 0x7f2e5c100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 -- 172.21.15.78:0/844968610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e5c103ee0 msgr2=0x7f2e5c101310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:25.064 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e5c103ee0 0x7f2e5c101310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 -- 172.21.15.78:0/844968610 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e4c0095f0 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e62161700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e5c0fee40 0x7f2e5c100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e61960700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c0ff7e0 0x7f2e5c100d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000d980 tx=0x7f2e5000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:25.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e5b7fe700 1 -- 172.21.15.78:0/844968610 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e50009b40 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e5c1a46b0 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e5b7fe700 1 -- 172.21.15.78:0/844968610 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2e5000de60 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.053+0000 7f2e5b7fe700 1 -- 172.21.15.78:0/844968610 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e5000f620 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.054+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e5c1a4c00 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.054+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e5c04e340 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.055+0000 7f2e5b7fe700 1 -- 172.21.15.78:0/844968610 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2e50010460 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.056+0000 7f2e5b7fe700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2e440a11e0 0x7f2e440a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.056+0000 7f2e62161700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2e440a11e0 0x7f2e440a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:25.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.056+0000 7f2e5b7fe700 1 -- 172.21.15.78:0/844968610 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f2e500cc800 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.056+0000 7f2e62161700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2e440a11e0 0x7f2e440a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2e54009dd0 tx=0x7f2e54009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:25.068 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.061+0000 7f2e5b7fe700 1 -- 172.21.15.78:0/844968610 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2e50095c00 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.194 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:25 smithi078 ceph-mon[127425]: pgmap v282: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-07-14T19:03:25.195 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:25 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1470172467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:03:25.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.258+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f2e5c1a4e20 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.259+0000 7f2e5b7fe700 1 -- 172.21.15.78:0/844968610 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2e50099220 con 0x7f2e5c0ff7e0 2023-07-14T19:03:25.263 INFO:teuthology.orchestra.run.smithi078.stdout:107374182455 2023-07-14T19:03:25.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2e440a11e0 msgr2=0x7f2e440a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:25.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2e440a11e0 0x7f2e440a3690 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2e54009dd0 tx=0x7f2e54009040 comp rx=0 tx=0).stop 2023-07-14T19:03:25.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c0ff7e0 msgr2=0x7f2e5c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:25.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c0ff7e0 0x7f2e5c100d90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f2e5000d980 tx=0x7f2e5000dd40 comp rx=0 tx=0).stop 2023-07-14T19:03:25.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 shutdown_connections 2023-07-14T19:03:25.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2e440a11e0 0x7f2e440a3690 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2e5c103ee0 0x7f2e5c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2e5c0fee40 0x7f2e5c100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 --2- 172.21.15.78:0/844968610 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2e5c0ff7e0 0x7f2e5c100d90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:25.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 >> 172.21.15.78:0/844968610 conn(0x7f2e5c075ca0 msgr2=0x7f2e5c102990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:25.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 shutdown_connections 2023-07-14T19:03:25.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:25.261+0000 7f2e643c5700 1 -- 172.21.15.78:0/844968610 wait complete. 2023-07-14T19:03:25.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:25 smithi088 ceph-mon[128973]: pgmap v282: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-07-14T19:03:25.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:25 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1470172467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:03:25.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:25 smithi102 ceph-mon[129280]: pgmap v282: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2023-07-14T19:03:25.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:25 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1470172467' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2023-07-14T19:03:25.807 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:26.067 INFO:tasks.cephadm.ceph_manager.ceph:need seq 296352743447 got 296352743450 for osd.10 2023-07-14T19:03:26.067 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:26.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 -- 172.21.15.78:0/1087429388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400febc0 msgr2=0x7f864010cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1087429388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400febc0 0x7f864010cb30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f863800b0a0 tx=0x7f863800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 -- 172.21.15.78:0/1087429388 shutdown_connections 2023-07-14T19:03:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1087429388 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8640104c70 0x7f8640105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1087429388 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400febc0 0x7f864010cb30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1087429388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86400fe210 0x7f86400fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 -- 172.21.15.78:0/1087429388 >> 172.21.15.78:0/1087429388 conn(0x7f8640075280 msgr2=0x7f8640075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 -- 172.21.15.78:0/1087429388 shutdown_connections 2023-07-14T19:03:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.049+0000 7f8644ac7700 1 -- 172.21.15.78:0/1087429388 wait complete. 2023-07-14T19:03:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f8644ac7700 1 Processor -- start 2023-07-14T19:03:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f8644ac7700 1 -- start start 2023-07-14T19:03:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f8644ac7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400fe210 0x7f8640195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f8644ac7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86400febc0 0x7f8640196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f8644ac7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8640104c70 0x7f864019a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:26.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f8644ac7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864019ad90 con 0x7f86400fe210 2023-07-14T19:03:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f8644ac7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864019af00 con 0x7f86400febc0 2023-07-14T19:03:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f8644ac7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f864019b0c0 con 0x7f8640104c70 2023-07-14T19:03:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f863ed9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8640104c70 0x7f864019a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:26.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f863e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400fe210 0x7f8640195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f863e59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400fe210 0x7f8640195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35376/0 (socket says 172.21.15.78:35376) 2023-07-14T19:03:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f863dd9b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86400febc0 0x7f8640196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.050+0000 7f863e59c700 1 -- 172.21.15.78:0/1706928555 learned_addr learned my addr 172.21.15.78:0/1706928555 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f863e59c700 1 -- 172.21.15.78:0/1706928555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8640104c70 msgr2=0x7f864019a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:26.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f863e59c700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8640104c70 0x7f864019a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f863e59c700 1 -- 172.21.15.78:0/1706928555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86400febc0 msgr2=0x7f8640196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f863e59c700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86400febc0 0x7f8640196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f863e59c700 1 -- 172.21.15.78:0/1706928555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8638009d00 con 0x7f86400fe210 2023-07-14T19:03:26.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f863dd9b700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86400febc0 0x7f8640196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:03:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f863e59c700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400fe210 0x7f8640195dd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f863000d980 tx=0x7f863000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f862f7fe700 1 -- 172.21.15.78:0/1706928555 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630004220 con 0x7f86400fe210 2023-07-14T19:03:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f864019b3a0 con 0x7f86400fe210 2023-07-14T19:03:26.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f862f7fe700 1 -- 172.21.15.78:0/1706928555 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8630007640 con 0x7f86400fe210 2023-07-14T19:03:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f862f7fe700 1 -- 172.21.15.78:0/1706928555 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8630005660 con 0x7f86400fe210 2023-07-14T19:03:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.051+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86400ff4f0 con 0x7f86400fe210 2023-07-14T19:03:26.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.052+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f864004e340 con 0x7f86400fe210 2023-07-14T19:03:26.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.054+0000 7f862f7fe700 1 -- 172.21.15.78:0/1706928555 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8630004750 con 0x7f86400fe210 2023-07-14T19:03:26.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.054+0000 7f862f7fe700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f86280a11e0 0x7f86280a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:26.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.055+0000 7f862f7fe700 1 -- 172.21.15.78:0/1706928555 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f86300cc950 con 0x7f86400fe210 2023-07-14T19:03:26.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.055+0000 7f863dd9b700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f86280a11e0 0x7f86280a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:26.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.055+0000 7f863dd9b700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f86280a11e0 0x7f86280a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f86401973f0 tx=0x7f86340073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:26.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.059+0000 7f862f7fe700 1 -- 172.21.15.78:0/1706928555 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8630095d50 con 0x7f86400fe210 2023-07-14T19:03:26.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.257+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f86400ff710 con 0x7f86400fe210 2023-07-14T19:03:26.261 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.258+0000 7f862f7fe700 1 -- 172.21.15.78:0/1706928555 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f8630016020 con 0x7f86400fe210 2023-07-14T19:03:26.262 INFO:teuthology.orchestra.run.smithi078.stdout:141733920820 2023-07-14T19:03:26.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f86280a11e0 msgr2=0x7f86280a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f86280a11e0 0x7f86280a3690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f86401973f0 tx=0x7f86340073d0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400fe210 msgr2=0x7f8640195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400fe210 0x7f8640195dd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f863000d980 tx=0x7f863000dc90 comp rx=0 tx=0).stop 2023-07-14T19:03:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 shutdown_connections 2023-07-14T19:03:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f86280a11e0 0x7f86280a3690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8640104c70 0x7f864019a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.264 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f86400fe210 0x7f8640195dd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 --2- 172.21.15.78:0/1706928555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86400febc0 0x7f8640196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 >> 172.21.15.78:0/1706928555 conn(0x7f8640075280 msgr2=0x7f86400fcaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 shutdown_connections 2023-07-14T19:03:26.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:26.260+0000 7f8644ac7700 1 -- 172.21.15.78:0/1706928555 wait complete. 2023-07-14T19:03:26.314 INFO:tasks.cephadm.ceph_manager.ceph:need seq 214748364837 got 214748364839 for osd.7 2023-07-14T19:03:26.315 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:26.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:26 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/844968610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:03:26.515 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:26 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/844968610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:03:26.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:26 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/844968610' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-07-14T19:03:27.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:27 smithi088 ceph-mon[128973]: pgmap v283: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:03:27.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:27 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1706928555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:03:27.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:27 smithi078 ceph-mon[127425]: pgmap v283: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:03:27.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:27 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1706928555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:03:27.471 INFO:tasks.cephadm.ceph_manager.ceph:need seq 274877906972 got 274877906973 for osd.9 2023-07-14T19:03:27.472 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:27.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:27 smithi102 ceph-mon[129280]: pgmap v283: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:03:27.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:27 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1706928555' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-07-14T19:03:27.661 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920818 got 141733920820 for osd.4 2023-07-14T19:03:27.661 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:28.337 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182453 got 107374182455 for osd.3 2023-07-14T19:03:28.337 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:28.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 -- 172.21.15.78:0/110977257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c0febc0 msgr2=0x7fe32c10cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:28.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 --2- 172.21.15.78:0/110977257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c0febc0 0x7fe32c10cb30 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fe32400b0a0 tx=0x7fe32400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 -- 172.21.15.78:0/110977257 shutdown_connections 2023-07-14T19:03:28.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 --2- 172.21.15.78:0/110977257 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe32c104c70 0x7fe32c105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 --2- 172.21.15.78:0/110977257 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c0febc0 0x7fe32c10cb30 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 --2- 172.21.15.78:0/110977257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe32c0fe210 0x7fe32c0fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 -- 172.21.15.78:0/110977257 >> 172.21.15.78:0/110977257 conn(0x7fe32c075280 msgr2=0x7fe32c075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:28.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 -- 172.21.15.78:0/110977257 shutdown_connections 2023-07-14T19:03:28.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.638+0000 7fe33181d700 1 -- 172.21.15.78:0/110977257 wait complete. 2023-07-14T19:03:28.644 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe33181d700 1 Processor -- start 2023-07-14T19:03:28.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe33181d700 1 -- start start 2023-07-14T19:03:28.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe33181d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe32c0fe210 0x7fe32c195d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:28.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe33181d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe32c0febc0 0x7fe32c1962c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:28.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe33181d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c104c70 0x7fe32c19a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:28.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe33181d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c19ad40 con 0x7fe32c104c70 2023-07-14T19:03:28.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe33181d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c19aeb0 con 0x7fe32c0fe210 2023-07-14T19:03:28.645 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe33181d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe32c19b070 con 0x7fe32c0febc0 2023-07-14T19:03:28.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe32b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c104c70 0x7fe32c19a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:28.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe32b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c104c70 0x7fe32c19a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35394/0 (socket says 172.21.15.78:35394) 2023-07-14T19:03:28.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe32b7fe700 1 -- 172.21.15.78:0/2069388546 learned_addr learned my addr 172.21.15.78:0/2069388546 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:28.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe32a7fc700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe32c0febc0 0x7fe32c1962c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:28.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.639+0000 7fe32affd700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe32c0fe210 0x7fe32c195d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:28.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe32b7fe700 1 -- 172.21.15.78:0/2069388546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe32c0febc0 msgr2=0x7fe32c1962c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:28.646 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe32b7fe700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe32c0febc0 0x7fe32c1962c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe32b7fe700 1 -- 172.21.15.78:0/2069388546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe32c0fe210 msgr2=0x7fe32c195d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe32b7fe700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe32c0fe210 0x7fe32c195d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe32b7fe700 1 -- 172.21.15.78:0/2069388546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe324009d00 con 0x7fe32c104c70 2023-07-14T19:03:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe32b7fe700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c104c70 0x7fe32c19a730 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe32400b780 tx=0x7fe324008f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe33081b700 1 -- 172.21.15.78:0/2069388546 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32400e040 con 0x7fe32c104c70 2023-07-14T19:03:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe33081b700 1 -- 172.21.15.78:0/2069388546 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe324004c50 con 0x7fe32c104c70 2023-07-14T19:03:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe32c19b2f0 con 0x7fe32c104c70 2023-07-14T19:03:28.647 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe33081b700 1 -- 172.21.15.78:0/2069388546 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3240052a0 con 0x7fe32c104c70 2023-07-14T19:03:28.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.640+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe32c1a2bb0 con 0x7fe32c104c70 2023-07-14T19:03:28.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.641+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe32c04e340 con 0x7fe32c104c70 2023-07-14T19:03:28.648 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.642+0000 7fe33081b700 1 -- 172.21.15.78:0/2069388546 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe32401d470 con 0x7fe32c104c70 2023-07-14T19:03:28.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.642+0000 7fe33081b700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe3140a11e0 0x7fe3140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:28.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.643+0000 7fe32affd700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe3140a11e0 0x7fe3140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:28.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.643+0000 7fe33081b700 1 -- 172.21.15.78:0/2069388546 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fe3240ce9e0 con 0x7fe32c104c70 2023-07-14T19:03:28.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.643+0000 7fe32affd700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe3140a11e0 0x7fe3140a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe320009bf0 tx=0x7fe320009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:28.649 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.646+0000 7fe33081b700 1 -- 172.21.15.78:0/2069388546 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe324097e70 con 0x7fe32c104c70 2023-07-14T19:03:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.839+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fe32c1a2d40 con 0x7fe32c104c70 2023-07-14T19:03:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.839+0000 7fe33081b700 1 -- 172.21.15.78:0/2069388546 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe32409b490 con 0x7fe32c104c70 2023-07-14T19:03:28.843 INFO:teuthology.orchestra.run.smithi078.stdout:163208757296 2023-07-14T19:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe3140a11e0 msgr2=0x7fe3140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe3140a11e0 0x7fe3140a3690 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe320009bf0 tx=0x7fe320009260 comp rx=0 tx=0).stop 2023-07-14T19:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c104c70 msgr2=0x7fe32c19a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c104c70 0x7fe32c19a730 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe32400b780 tx=0x7fe324008f90 comp rx=0 tx=0).stop 2023-07-14T19:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 shutdown_connections 2023-07-14T19:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe3140a11e0 0x7fe3140a3690 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe32c0fe210 0x7fe32c195d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe32c104c70 0x7fe32c19a730 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 --2- 172.21.15.78:0/2069388546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe32c0febc0 0x7fe32c1962c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 >> 172.21.15.78:0/2069388546 conn(0x7fe32c075280 msgr2=0x7fe32c0fcad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 shutdown_connections 2023-07-14T19:03:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.841+0000 7fe33181d700 1 -- 172.21.15.78:0/2069388546 wait complete. 2023-07-14T19:03:28.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 -- 172.21.15.78:0/1173760148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0ff7e0 msgr2=0x7f196c1037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:28.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 --2- 172.21.15.78:0/1173760148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0ff7e0 0x7f196c1037b0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f19540099c0 tx=0x7f1954009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 -- 172.21.15.78:0/1173760148 shutdown_connections 2023-07-14T19:03:28.954 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 --2- 172.21.15.78:0/1173760148 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f196c103ee0 0x7f196c10ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 --2- 172.21.15.78:0/1173760148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f196c0fee40 0x7f196c0ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 --2- 172.21.15.78:0/1173760148 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0ff7e0 0x7f196c1037b0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 -- 172.21.15.78:0/1173760148 >> 172.21.15.78:0/1173760148 conn(0x7f196c075ca0 msgr2=0x7f196c0760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:28.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 -- 172.21.15.78:0/1173760148 shutdown_connections 2023-07-14T19:03:28.955 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.950+0000 7f1971cf5700 1 -- 172.21.15.78:0/1173760148 wait complete. 2023-07-14T19:03:28.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f1971cf5700 1 Processor -- start 2023-07-14T19:03:28.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f1971cf5700 1 -- start start 2023-07-14T19:03:28.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f1971cf5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0fee40 0x7f196c196ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:28.957 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f1971cf5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f196c0ff7e0 0x7f196c1970e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:28.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f1971cf5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f196c103ee0 0x7f196c1a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:28.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f1971cf5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c197a20 con 0x7f196c0fee40 2023-07-14T19:03:28.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f1971cf5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c1a2160 con 0x7f196c0ff7e0 2023-07-14T19:03:28.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f1971cf5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f196c1a2320 con 0x7f196c103ee0 2023-07-14T19:03:28.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f196b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0fee40 0x7f196c196ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:28.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f196b7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0fee40 0x7f196c196ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35404/0 (socket says 172.21.15.78:35404) 2023-07-14T19:03:28.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f196b7fe700 1 -- 172.21.15.78:0/2166342825 learned_addr learned my addr 172.21.15.78:0/2166342825 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:28.958 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f196affd700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f196c0ff7e0 0x7f196c1970e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:28.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.951+0000 7f196bfff700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f196c103ee0 0x7f196c1a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:28.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f196b7fe700 1 -- 172.21.15.78:0/2166342825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f196c103ee0 msgr2=0x7f196c1a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:28.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f196b7fe700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f196c103ee0 0x7f196c1a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f196b7fe700 1 -- 172.21.15.78:0/2166342825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f196c0ff7e0 msgr2=0x7f196c1970e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:28.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f196b7fe700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f196c0ff7e0 0x7f196c1970e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:28.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f196b7fe700 1 -- 172.21.15.78:0/2166342825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19540095f0 con 0x7f196c0fee40 2023-07-14T19:03:28.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f196b7fe700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0fee40 0x7f196c196ba0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f195c00e9a0 tx=0x7f195c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:28.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f1968ff9700 1 -- 172.21.15.78:0/2166342825 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c009790 con 0x7f196c0fee40 2023-07-14T19:03:28.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f1968ff9700 1 -- 172.21.15.78:0/2166342825 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f195c00cbd0 con 0x7f196c0fee40 2023-07-14T19:03:28.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f196c1a2600 con 0x7f196c0fee40 2023-07-14T19:03:28.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f1968ff9700 1 -- 172.21.15.78:0/2166342825 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f195c018680 con 0x7f196c0fee40 2023-07-14T19:03:28.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.952+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f196c1a2ad0 con 0x7f196c0fee40 2023-07-14T19:03:28.960 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.953+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f196c04e340 con 0x7f196c0fee40 2023-07-14T19:03:28.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.954+0000 7f1968ff9700 1 -- 172.21.15.78:0/2166342825 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f195c00cd40 con 0x7f196c0fee40 2023-07-14T19:03:28.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.955+0000 7f1968ff9700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f19580a11e0 0x7f19580a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:28.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.955+0000 7f196affd700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f19580a11e0 0x7f19580a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:28.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.955+0000 7f1968ff9700 1 -- 172.21.15.78:0/2166342825 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f195c014070 con 0x7f196c0fee40 2023-07-14T19:03:28.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.955+0000 7f196affd700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f19580a11e0 0x7f19580a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f196c1037f0 tx=0x7f195401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:28.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:28.958+0000 7f1968ff9700 1 -- 172.21.15.78:0/2166342825 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f195c0965e0 con 0x7f196c0fee40 2023-07-14T19:03:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.173+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f196c1a2e20 con 0x7f196c0fee40 2023-07-14T19:03:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.174+0000 7f1968ff9700 1 -- 172.21.15.78:0/2166342825 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f195c026090 con 0x7f196c0fee40 2023-07-14T19:03:29.178 INFO:teuthology.orchestra.run.smithi078.stdout:34359738438 2023-07-14T19:03:29.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f19580a11e0 msgr2=0x7f19580a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:29.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f19580a11e0 0x7f19580a3690 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f196c1037f0 tx=0x7f195401a040 comp rx=0 tx=0).stop 2023-07-14T19:03:29.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0fee40 msgr2=0x7f196c196ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:29.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0fee40 0x7f196c196ba0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f195c00e9a0 tx=0x7f195c00ed60 comp rx=0 tx=0).stop 2023-07-14T19:03:29.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 shutdown_connections 2023-07-14T19:03:29.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f19580a11e0 0x7f19580a3690 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.180 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f196c103ee0 0x7f196c1a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f196c0fee40 0x7f196c196ba0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 --2- 172.21.15.78:0/2166342825 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f196c0ff7e0 0x7f196c1970e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 >> 172.21.15.78:0/2166342825 conn(0x7f196c075ca0 msgr2=0x7f196c0fc4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:29.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 shutdown_connections 2023-07-14T19:03:29.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.176+0000 7f1971cf5700 1 -- 172.21.15.78:0/2166342825 wait complete. 2023-07-14T19:03:29.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 -- 172.21.15.78:0/906757513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c0fff30 msgr2=0x7f900c100300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:29.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 --2- 172.21.15.78:0/906757513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c0fff30 0x7f900c100300 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f90000099c0 tx=0x7f9000009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 -- 172.21.15.78:0/906757513 shutdown_connections 2023-07-14T19:03:29.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 --2- 172.21.15.78:0/906757513 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f900c104fd0 0x7f900c10cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 --2- 172.21.15.78:0/906757513 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c0fff30 0x7f900c100300 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 --2- 172.21.15.78:0/906757513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900c1008d0 0x7f900c1048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 -- 172.21.15.78:0/906757513 >> 172.21.15.78:0/906757513 conn(0x7f900c0fb790 msgr2=0x7f900c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:29.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 -- 172.21.15.78:0/906757513 shutdown_connections 2023-07-14T19:03:29.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 -- 172.21.15.78:0/906757513 wait complete. 2023-07-14T19:03:29.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.233+0000 7f90121fb700 1 Processor -- start 2023-07-14T19:03:29.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f90121fb700 1 -- start start 2023-07-14T19:03:29.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f90121fb700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f900c0fff30 0x7f900c19ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:29.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f90121fb700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900c1008d0 0x7f900c19bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:29.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f90121fb700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c104fd0 0x7f900c195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:29.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f90121fb700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c196110 con 0x7f900c104fd0 2023-07-14T19:03:29.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f90121fb700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c196280 con 0x7f900c0fff30 2023-07-14T19:03:29.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f90121fb700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f900c196440 con 0x7f900c1008d0 2023-07-14T19:03:29.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c104fd0 0x7f900c195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:29.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900b7fe700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f900c0fff30 0x7f900c19ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:29.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c104fd0 0x7f900c195aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35422/0 (socket says 172.21.15.78:35422) 2023-07-14T19:03:29.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900bfff700 1 -- 172.21.15.78:0/2019199078 learned_addr learned my addr 172.21.15.78:0/2019199078 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:29.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900affd700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900c1008d0 0x7f900c19bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:29.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900bfff700 1 -- 172.21.15.78:0/2019199078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900c1008d0 msgr2=0x7f900c19bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:29.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900bfff700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900c1008d0 0x7f900c19bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900bfff700 1 -- 172.21.15.78:0/2019199078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f900c0fff30 msgr2=0x7f900c19ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:29.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900bfff700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f900c0fff30 0x7f900c19ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.234+0000 7f900bfff700 1 -- 172.21.15.78:0/2019199078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90000095f0 con 0x7f900c104fd0 2023-07-14T19:03:29.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.235+0000 7f900b7fe700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f900c0fff30 0x7f900c19ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:03:29.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.235+0000 7f900bfff700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c104fd0 0x7f900c195aa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f900400ada0 tx=0x7f900400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:29.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.235+0000 7f9008ff9700 1 -- 172.21.15.78:0/2019199078 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f900400f070 con 0x7f900c104fd0 2023-07-14T19:03:29.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.235+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f900c196720 con 0x7f900c104fd0 2023-07-14T19:03:29.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.235+0000 7f9008ff9700 1 -- 172.21.15.78:0/2019199078 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9004007b30 con 0x7f900c104fd0 2023-07-14T19:03:29.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.235+0000 7f9008ff9700 1 -- 172.21.15.78:0/2019199078 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9004005280 con 0x7f900c104fd0 2023-07-14T19:03:29.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.235+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f900c1a2a10 con 0x7f900c104fd0 2023-07-14T19:03:29.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.236+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f900c04e340 con 0x7f900c104fd0 2023-07-14T19:03:29.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.237+0000 7f9008ff9700 1 -- 172.21.15.78:0/2019199078 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f900400c980 con 0x7f900c104fd0 2023-07-14T19:03:29.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.237+0000 7f9008ff9700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ff40a11e0 0x7f8ff40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:29.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.238+0000 7f9008ff9700 1 -- 172.21.15.78:0/2019199078 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7f90040cc2b0 con 0x7f900c104fd0 2023-07-14T19:03:29.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.238+0000 7f900b7fe700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ff40a11e0 0x7f8ff40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:29.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.238+0000 7f900b7fe700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ff40a11e0 0x7f8ff40a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9000000c00 tx=0x7f9000009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:29.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.242+0000 7f9008ff9700 1 -- 172.21.15.78:0/2019199078 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9004095740 con 0x7f900c104fd0 2023-07-14T19:03:29.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.439+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f900c1a2c00 con 0x7f900c104fd0 2023-07-14T19:03:29.443 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.440+0000 7f9008ff9700 1 -- 172.21.15.78:0/2019199078 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f900400cc30 con 0x7f900c104fd0 2023-07-14T19:03:29.444 INFO:teuthology.orchestra.run.smithi078.stdout:193273528364 2023-07-14T19:03:29.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ff40a11e0 msgr2=0x7f8ff40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:29.446 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ff40a11e0 0x7f8ff40a3690 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f9000000c00 tx=0x7f9000009f10 comp rx=0 tx=0).stop 2023-07-14T19:03:29.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c104fd0 msgr2=0x7f900c195aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:29.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c104fd0 0x7f900c195aa0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f900400ada0 tx=0x7f900400c330 comp rx=0 tx=0).stop 2023-07-14T19:03:29.447 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 shutdown_connections 2023-07-14T19:03:29.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ff40a11e0 0x7f8ff40a3690 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f900c0fff30 0x7f900c19ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.448 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f900c104fd0 0x7f900c195aa0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 --2- 172.21.15.78:0/2019199078 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f900c1008d0 0x7f900c19bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:29.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 >> 172.21.15.78:0/2019199078 conn(0x7f900c0fb790 msgr2=0x7f900c103880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:29.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 shutdown_connections 2023-07-14T19:03:29.449 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:29.442+0000 7f90121fb700 1 -- 172.21.15.78:0/2019199078 wait complete. 2023-07-14T19:03:29.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:29 smithi088 ceph-mon[128973]: pgmap v284: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:03:29.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:29 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2069388546' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:03:29.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:29 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2166342825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:03:29.472 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:29 smithi078 ceph-mon[127425]: pgmap v284: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:03:29.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:29 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2069388546' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:03:29.473 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:29 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2166342825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:03:29.514 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757294 got 163208757296 for osd.5 2023-07-14T19:03:29.514 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:29.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:29 smithi102 ceph-mon[129280]: pgmap v284: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:03:29.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:29 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2069388546' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-07-14T19:03:29.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:29 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2166342825' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-07-14T19:03:29.961 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738435 got 34359738438 for osd.0 2023-07-14T19:03:29.961 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:30.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 -- 172.21.15.78:0/1871454889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c0fef40 msgr2=0x7fbc6c0ff310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:30.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 --2- 172.21.15.78:0/1871454889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c0fef40 0x7fbc6c0ff310 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fbc600099c0 tx=0x7fbc60009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 -- 172.21.15.78:0/1871454889 shutdown_connections 2023-07-14T19:03:30.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 --2- 172.21.15.78:0/1871454889 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc6c101150 0x7fbc6c10bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 --2- 172.21.15.78:0/1871454889 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c0fef40 0x7fbc6c0ff310 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 --2- 172.21.15.78:0/1871454889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc6c1007a0 0x7fbc6c100c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 -- 172.21.15.78:0/1871454889 >> 172.21.15.78:0/1871454889 conn(0x7fbc6c075c90 msgr2=0x7fbc6c076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 -- 172.21.15.78:0/1871454889 shutdown_connections 2023-07-14T19:03:30.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.104+0000 7fbc72b66700 1 -- 172.21.15.78:0/1871454889 wait complete. 2023-07-14T19:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc72b66700 1 Processor -- start 2023-07-14T19:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc72b66700 1 -- start start 2023-07-14T19:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc72b66700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc6c0fef40 0x7fbc6c196b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc72b66700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c1007a0 0x7fbc6c1970b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc72b66700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc6c101150 0x7fbc6c1a1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:30.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc72b66700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c1979c0 con 0x7fbc6c1007a0 2023-07-14T19:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc72b66700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c1a2130 con 0x7fbc6c101150 2023-07-14T19:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc72b66700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc6c1a22a0 con 0x7fbc6c0fef40 2023-07-14T19:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c1007a0 0x7fbc6c1970b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc6bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c1007a0 0x7fbc6c1970b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35434/0 (socket says 172.21.15.78:35434) 2023-07-14T19:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc6bfff700 1 -- 172.21.15.78:0/2286332646 learned_addr learned my addr 172.21.15.78:0/2286332646 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.105+0000 7fbc70902700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc6c0fef40 0x7fbc6c196b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc71103700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc6c101150 0x7fbc6c1a1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:30.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc6bfff700 1 -- 172.21.15.78:0/2286332646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc6c0fef40 msgr2=0x7fbc6c196b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc6bfff700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc6c0fef40 0x7fbc6c196b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc6bfff700 1 -- 172.21.15.78:0/2286332646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc6c101150 msgr2=0x7fbc6c1a1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc6bfff700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc6c101150 0x7fbc6c1a1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc6bfff700 1 -- 172.21.15.78:0/2286332646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc600095f0 con 0x7fbc6c1007a0 2023-07-14T19:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc6bfff700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c1007a0 0x7fbc6c1970b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c920 tx=0x7fbc5800cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc69ffb700 1 -- 172.21.15.78:0/2286332646 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58007b00 con 0x7fbc6c1007a0 2023-07-14T19:03:30.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc69ffb700 1 -- 172.21.15.78:0/2286332646 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fbc5800ce70 con 0x7fbc6c1007a0 2023-07-14T19:03:30.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc69ffb700 1 -- 172.21.15.78:0/2286332646 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc58005440 con 0x7fbc6c1007a0 2023-07-14T19:03:30.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc6c1a2580 con 0x7fbc6c1007a0 2023-07-14T19:03:30.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.106+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc6c1a2a30 con 0x7fbc6c1007a0 2023-07-14T19:03:30.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.107+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc6c04e340 con 0x7fbc6c1007a0 2023-07-14T19:03:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.109+0000 7fbc69ffb700 1 -- 172.21.15.78:0/2286332646 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fbc580055a0 con 0x7fbc6c1007a0 2023-07-14T19:03:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.109+0000 7fbc69ffb700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbc5c0a1260 0x7fbc5c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.109+0000 7fbc70902700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbc5c0a1260 0x7fbc5c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.109+0000 7fbc69ffb700 1 -- 172.21.15.78:0/2286332646 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8730+0+0 (secure 0 0 0) 0x7fbc580ccc20 con 0x7fbc6c1007a0 2023-07-14T19:03:30.115 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.110+0000 7fbc70902700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbc5c0a1260 0x7fbc5c0a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b580 tx=0x7fbc60005c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:30.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.115+0000 7fbc69ffb700 1 -- 172.21.15.78:0/2286332646 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fbc580960a0 con 0x7fbc6c1007a0 2023-07-14T19:03:30.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.312+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7fbc6c068f80 con 0x7fbc6c1007a0 2023-07-14T19:03:30.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.313+0000 7fbc69ffb700 1 -- 172.21.15.78:0/2286332646 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7fbc5801d070 con 0x7fbc6c1007a0 2023-07-14T19:03:30.317 INFO:teuthology.orchestra.run.smithi078.stdout:326417514519 2023-07-14T19:03:30.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbc5c0a1260 msgr2=0x7fbc5c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbc5c0a1260 0x7fbc5c0a3710 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fbc6000b580 tx=0x7fbc60005c00 comp rx=0 tx=0).stop 2023-07-14T19:03:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c1007a0 msgr2=0x7fbc6c1970b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c1007a0 0x7fbc6c1970b0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fbc5800c920 tx=0x7fbc5800cc30 comp rx=0 tx=0).stop 2023-07-14T19:03:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 shutdown_connections 2023-07-14T19:03:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fbc5c0a1260 0x7fbc5c0a3710 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.319 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fbc6c101150 0x7fbc6c1a1bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc6c0fef40 0x7fbc6c196b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 --2- 172.21.15.78:0/2286332646 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fbc6c1007a0 0x7fbc6c1970b0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:30.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 >> 172.21.15.78:0/2286332646 conn(0x7fbc6c075c90 msgr2=0x7fbc6c0fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:30.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 shutdown_connections 2023-07-14T19:03:30.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:30.315+0000 7fbc72b66700 1 -- 172.21.15.78:0/2286332646 wait complete. 2023-07-14T19:03:30.369 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528362 got 193273528364 for osd.6 2023-07-14T19:03:30.369 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:30.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2019199078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:03:30.504 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-07-14T19:03:30.504 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2023-07-14T19:03:30.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-07-14T19:03:30.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-07-14T19:03:30.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-07-14T19:03:30.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-07-14T19:03:30.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2023-07-14T19:03:30.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2023-07-14T19:03:30.505 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2023-07-14T19:03:30.506 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2023-07-14T19:03:30.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2019199078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:03:30.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-07-14T19:03:30.506 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2023-07-14T19:03:30.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-07-14T19:03:30.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-07-14T19:03:30.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-07-14T19:03:30.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-07-14T19:03:30.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2023-07-14T19:03:30.507 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2023-07-14T19:03:30.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2023-07-14T19:03:30.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2023-07-14T19:03:30.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2019199078' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-07-14T19:03:30.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2023-07-14T19:03:30.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2023-07-14T19:03:30.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]: dispatch 2023-07-14T19:03:30.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2023-07-14T19:03:30.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]: dispatch 2023-07-14T19:03:30.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2023-07-14T19:03:30.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]: dispatch 2023-07-14T19:03:30.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]: dispatch 2023-07-14T19:03:30.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]: dispatch 2023-07-14T19:03:30.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]: dispatch 2023-07-14T19:03:30.717 INFO:tasks.cephadm.ceph_manager.ceph:need seq 326417514517 got 326417514519 for osd.11 2023-07-14T19:03:30.718 DEBUG:teuthology.parallel:result is None 2023-07-14T19:03:30.718 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-07-14T19:03:30.718 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:03:31.271 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:31.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: pgmap v285: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:03:31.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-07-14T19:03:31.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2023-07-14T19:03:31.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-07-14T19:03:31.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-07-14T19:03:31.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-07-14T19:03:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-07-14T19:03:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2023-07-14T19:03:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2023-07-14T19:03:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2023-07-14T19:03:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2023-07-14T19:03:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T19:03:31.463 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:31 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2286332646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:03:31.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: pgmap v285: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:03:31.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-07-14T19:03:31.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2023-07-14T19:03:31.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-07-14T19:03:31.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-07-14T19:03:31.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-07-14T19:03:31.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-07-14T19:03:31.549 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2023-07-14T19:03:31.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2023-07-14T19:03:31.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2023-07-14T19:03:31.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2023-07-14T19:03:31.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T19:03:31.550 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:31 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2286332646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:03:31.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: pgmap v285: 129 pgs: 129 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:03:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2023-07-14T19:03:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2023-07-14T19:03:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1a", "id": [8, 1]}]': finished 2023-07-14T19:03:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2023-07-14T19:03:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8]}]': finished 2023-07-14T19:03:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2023-07-14T19:03:31.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [9, 10]}]': finished 2023-07-14T19:03:31.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [5, 10]}]': finished 2023-07-14T19:03:31.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.16", "id": [9, 8]}]': finished 2023-07-14T19:03:31.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.18", "id": [9, 8]}]': finished 2023-07-14T19:03:31.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: osdmap e86: 12 total, 12 up, 12 in 2023-07-14T19:03:31.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:31 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2286332646' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2023-07-14T19:03:32.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:32 smithi078 ceph-mon[127425]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T19:03:32.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:32 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:03:32.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:32 smithi102 ceph-mon[129280]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T19:03:32.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:32 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:03:32.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:32 smithi088 ceph-mon[128973]: osdmap e87: 12 total, 12 up, 12 in 2023-07-14T19:03:32.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:32 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:03:32.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/2379477204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 msgr2=0x7f8a9810c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:32.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/2379477204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 0x7f8a9810c4f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f8a9000b0a0 tx=0x7f8a9000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:32.787 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/2379477204 shutdown_connections 2023-07-14T19:03:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/2379477204 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a98100d40 0x7f8a98101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/2379477204 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 0x7f8a9810c4f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/2379477204 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a98101650 0x7f8a98109b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/2379477204 >> 172.21.15.78:0/2379477204 conn(0x7f8a980fa580 msgr2=0x7f8a980fc990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/2379477204 shutdown_connections 2023-07-14T19:03:32.788 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.783+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/2379477204 wait complete. 2023-07-14T19:03:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.784+0000 7f8a9ef9e700 1 Processor -- start 2023-07-14T19:03:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.784+0000 7f8a9ef9e700 1 -- start start 2023-07-14T19:03:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.784+0000 7f8a9ef9e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a98100d40 0x7f8a980fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:32.790 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.784+0000 7f8a9ef9e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a98101650 0x7f8a980feee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.784+0000 7f8a9ef9e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 0x7f8a980ff460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.784+0000 7f8a9ef9e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a981a4210 con 0x7f8a9810a090 2023-07-14T19:03:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.784+0000 7f8a9ef9e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a981a4380 con 0x7f8a98100d40 2023-07-14T19:03:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.784+0000 7f8a9ef9e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a981a4540 con 0x7f8a98101650 2023-07-14T19:03:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 0x7f8a980ff460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 0x7f8a980ff460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35448/0 (socket says 172.21.15.78:35448) 2023-07-14T19:03:32.791 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 -- 172.21.15.78:0/3717169544 learned_addr learned my addr 172.21.15.78:0/3717169544 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:32.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9df9c700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a98100d40 0x7f8a980fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:32.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9d79b700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a98101650 0x7f8a980feee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:32.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 -- 172.21.15.78:0/3717169544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a98101650 msgr2=0x7f8a980feee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:32.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a98101650 0x7f8a980feee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:32.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 -- 172.21.15.78:0/3717169544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a98100d40 msgr2=0x7f8a980fe9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:32.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a98100d40 0x7f8a980fe9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:32.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 -- 172.21.15.78:0/3717169544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a90009d00 con 0x7f8a9810a090 2023-07-14T19:03:32.792 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9d79b700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a98101650 0x7f8a980feee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9e79d700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 0x7f8a980ff460 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8a900062a0 tx=0x7f8a90004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a8effd700 1 -- 172.21.15.78:0/3717169544 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9000e040 con 0x7f8a9810a090 2023-07-14T19:03:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a981a47c0 con 0x7f8a9810a090 2023-07-14T19:03:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a8effd700 1 -- 172.21.15.78:0/3717169544 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8a90003e90 con 0x7f8a9810a090 2023-07-14T19:03:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a8effd700 1 -- 172.21.15.78:0/3717169544 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a90004020 con 0x7f8a9810a090 2023-07-14T19:03:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.785+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a981a4d10 con 0x7f8a9810a090 2023-07-14T19:03:32.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.786+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a98105250 con 0x7f8a9810a090 2023-07-14T19:03:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.788+0000 7f8a8effd700 1 -- 172.21.15.78:0/3717169544 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8a90003a00 con 0x7f8a9810a090 2023-07-14T19:03:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.788+0000 7f8a8effd700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a840a11e0 0x7f8a840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:32.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.789+0000 7f8a9df9c700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a840a11e0 0x7f8a840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.789+0000 7f8a8effd700 1 -- 172.21.15.78:0/3717169544 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8a900ceb20 con 0x7f8a9810a090 2023-07-14T19:03:32.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.789+0000 7f8a9df9c700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a840a11e0 0x7f8a840a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009a10 tx=0x7f8a88009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:32.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.792+0000 7f8a8effd700 1 -- 172.21.15.78:0/3717169544 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8a90097e90 con 0x7f8a9810a090 2023-07-14T19:03:32.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.985+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8a9804eb90 con 0x7f8a840a11e0 2023-07-14T19:03:33.002 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:32.999+0000 7f8a8effd700 1 -- 172.21.15.78:0/3717169544 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+423694 (secure 0 0 0) 0x7f8a9804eb90 con 0x7f8a840a11e0 2023-07-14T19:03:33.007 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:33.009 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a840a11e0 msgr2=0x7f8a840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a840a11e0 0x7f8a840a3690 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f8a88009a10 tx=0x7f8a88009190 comp rx=0 tx=0).stop 2023-07-14T19:03:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 msgr2=0x7f8a980ff460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 0x7f8a980ff460 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f8a900062a0 tx=0x7f8a90004d00 comp rx=0 tx=0).stop 2023-07-14T19:03:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 shutdown_connections 2023-07-14T19:03:33.010 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8a840a11e0 0x7f8a840a3690 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8a98100d40 0x7f8a980fe9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8a9810a090 0x7f8a980ff460 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 --2- 172.21.15.78:0/3717169544 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a98101650 0x7f8a980feee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 >> 172.21.15.78:0/3717169544 conn(0x7f8a980fa580 msgr2=0x7f8a980fb1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 shutdown_connections 2023-07-14T19:03:33.011 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:33.003+0000 7f8a9ef9e700 1 -- 172.21.15.78:0/3717169544 wait complete. 2023-07-14T19:03:33.012 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-14T19:03:33.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:33 smithi078 ceph-mon[127425]: pgmap v288: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:03:33.539 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2023-07-14T19:03:32.125375+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651878,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":12,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":390,"num_read_kb":303,"num_write":336,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337376,"kb_used_data":13156,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604448,"statfs":{"total":1151940427776,"available":1151594954752,"internally_reserved":0,"allocated":13471744,"data_stored":7487953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":12,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":44,"num_read_kb":19,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002205"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.209253+0000","last_change":"2023-07-14T19:02:30.582427+0000","last_active":"2023-07-14T19:03:30.209253+0000","last_peered":"2023-07-14T19:03:30.209253+0000","last_clean":"2023-07-14T19:03:30.209253+0000","last_became_active":"2023-07-14T19:02:30.582149+0000","last_became_peered":"2023-07-14T19:02:30.582149+0000","last_unstale":"2023-07-14T19:03:30.209253+0000","last_undegraded":"2023-07-14T19:03:30.209253+0000","last_fullsized":"2023-07-14T19:03:30.209253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:43:06.312716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208324+0000","last_change":"2023-07-14T19:02:30.567053+0000","last_active":"2023-07-14T19:03:31.208324+0000","last_peered":"2023-07-14T19:03:31.208324+0000","last_clean":"2023-07-14T19:03:31.208324+0000","last_became_active":"2023-07-14T19:02:30.566736+0000","last_became_peered":"2023-07-14T19:02:30.566736+0000","last_unstale":"2023-07-14T19:03:31.208324+0000","last_undegraded":"2023-07-14T19:03:31.208324+0000","last_fullsized":"2023-07-14T19:03:31.208324+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:44:23.099904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208244+0000","last_change":"2023-07-14T19:02:30.577632+0000","last_active":"2023-07-14T19:03:31.208244+0000","last_peered":"2023-07-14T19:03:31.208244+0000","last_clean":"2023-07-14T19:03:31.208244+0000","last_became_active":"2023-07-14T19:02:30.577300+0000","last_became_peered":"2023-07-14T19:02:30.577300+0000","last_unstale":"2023-07-14T19:03:31.208244+0000","last_undegraded":"2023-07-14T19:03:31.208244+0000","last_fullsized":"2023-07-14T19:03:31.208244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:15:18.165242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213866+0000","last_change":"2023-07-14T19:02:30.581405+0000","last_active":"2023-07-14T19:03:31.213866+0000","last_peered":"2023-07-14T19:03:31.213866+0000","last_clean":"2023-07-14T19:03:31.213866+0000","last_became_active":"2023-07-14T19:02:30.581113+0000","last_became_peered":"2023-07-14T19:02:30.581113+0000","last_unstale":"2023-07-14T19:03:31.213866+0000","last_undegraded":"2023-07-14T19:03:31.213866+0000","last_fullsized":"2023-07-14T19:03:31.213866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:42:53.074489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206313+0000","last_change":"2023-07-14T19:02:30.574285+0000","last_active":"2023-07-14T19:03:30.206313+0000","last_peered":"2023-07-14T19:03:30.206313+0000","last_clean":"2023-07-14T19:03:30.206313+0000","last_became_active":"2023-07-14T19:02:30.573967+0000","last_became_peered":"2023-07-14T19:02:30.573967+0000","last_unstale":"2023-07-14T19:03:30.206313+0000","last_undegraded":"2023-07-14T19:03:30.206313+0000","last_fullsized":"2023-07-14T19:03:30.206313+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:42:45.695145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034390+0000","last_change":"2023-07-14T19:02:30.574135+0000","last_active":"2023-07-14T19:02:36.034390+0000","last_peered":"2023-07-14T19:02:36.034390+0000","last_clean":"2023-07-14T19:02:36.034390+0000","last_became_active":"2023-07-14T19:02:30.573821+0000","last_became_peered":"2023-07-14T19:02:30.573821+0000","last_unstale":"2023-07-14T19:02:36.034390+0000","last_undegraded":"2023-07-14T19:02:36.034390+0000","last_fullsized":"2023-07-14T19:02:36.034390+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:35:07.847570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.213960+0000","last_change":"2023-07-14T19:02:30.578051+0000","last_active":"2023-07-14T19:03:30.213960+0000","last_peered":"2023-07-14T19:03:30.213960+0000","last_clean":"2023-07-14T19:03:30.213960+0000","last_became_active":"2023-07-14T19:02:30.577700+0000","last_became_peered":"2023-07-14T19:02:30.577700+0000","last_unstale":"2023-07-14T19:03:30.213960+0000","last_undegraded":"2023-07-14T19:03:30.213960+0000","last_fullsized":"2023-07-14T19:03:30.213960+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:57.113901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811504+0000","last_change":"2023-07-14T19:02:30.567480+0000","last_active":"2023-07-14T19:02:35.811504+0000","last_peered":"2023-07-14T19:02:35.811504+0000","last_clean":"2023-07-14T19:02:35.811504+0000","last_became_active":"2023-07-14T19:02:30.567147+0000","last_became_peered":"2023-07-14T19:02:30.567147+0000","last_unstale":"2023-07-14T19:02:35.811504+0000","last_undegraded":"2023-07-14T19:02:35.811504+0000","last_fullsized":"2023-07-14T19:02:35.811504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:27:53.531353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607383+0000","last_change":"2023-07-14T19:02:30.581936+0000","last_active":"2023-07-14T19:02:35.607383+0000","last_peered":"2023-07-14T19:02:35.607383+0000","last_clean":"2023-07-14T19:02:35.607383+0000","last_became_active":"2023-07-14T19:02:30.579619+0000","last_became_peered":"2023-07-14T19:02:30.579619+0000","last_unstale":"2023-07-14T19:02:35.607383+0000","last_undegraded":"2023-07-14T19:02:35.607383+0000","last_fullsized":"2023-07-14T19:02:35.607383+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:33:46.396910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.034647+0000","last_change":"2023-07-14T19:02:30.574975+0000","last_active":"2023-07-14T19:02:36.034647+0000","last_peered":"2023-07-14T19:02:36.034647+0000","last_clean":"2023-07-14T19:02:36.034647+0000","last_became_active":"2023-07-14T19:02:30.574633+0000","last_became_peered":"2023-07-14T19:02:30.574633+0000","last_unstale":"2023-07-14T19:02:36.034647+0000","last_undegraded":"2023-07-14T19:02:36.034647+0000","last_fullsized":"2023-07-14T19:02:36.034647+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:59:33.710502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214597+0000","last_change":"2023-07-14T19:02:30.583662+0000","last_active":"2023-07-14T19:03:30.214597+0000","last_peered":"2023-07-14T19:03:30.214597+0000","last_clean":"2023-07-14T19:03:30.214597+0000","last_became_active":"2023-07-14T19:02:30.583352+0000","last_became_peered":"2023-07-14T19:02:30.583352+0000","last_unstale":"2023-07-14T19:03:30.214597+0000","last_undegraded":"2023-07-14T19:03:30.214597+0000","last_fullsized":"2023-07-14T19:03:30.214597+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:11:29.149234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213712+0000","last_change":"2023-07-14T19:02:30.567685+0000","last_active":"2023-07-14T19:03:31.213712+0000","last_peered":"2023-07-14T19:03:31.213712+0000","last_clean":"2023-07-14T19:03:31.213712+0000","last_became_active":"2023-07-14T19:02:30.566674+0000","last_became_peered":"2023-07-14T19:02:30.566674+0000","last_unstale":"2023-07-14T19:03:31.213712+0000","last_undegraded":"2023-07-14T19:03:31.213712+0000","last_fullsized":"2023-07-14T19:03:31.213712+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:41:27.460945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608702+0000","last_change":"2023-07-14T19:02:30.558012+0000","last_active":"2023-07-14T19:02:35.608702+0000","last_peered":"2023-07-14T19:02:35.608702+0000","last_clean":"2023-07-14T19:02:35.608702+0000","last_became_active":"2023-07-14T19:02:30.557675+0000","last_became_peered":"2023-07-14T19:02:30.557675+0000","last_unstale":"2023-07-14T19:02:35.608702+0000","last_undegraded":"2023-07-14T19:02:35.608702+0000","last_fullsized":"2023-07-14T19:02:35.608702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:56.276352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207948+0000","last_change":"2023-07-14T19:02:30.570341+0000","last_active":"2023-07-14T19:03:31.207948+0000","last_peered":"2023-07-14T19:03:31.207948+0000","last_clean":"2023-07-14T19:03:31.207948+0000","last_became_active":"2023-07-14T19:02:30.570091+0000","last_became_peered":"2023-07-14T19:02:30.570091+0000","last_unstale":"2023-07-14T19:03:31.207948+0000","last_undegraded":"2023-07-14T19:03:31.207948+0000","last_fullsized":"2023-07-14T19:03:31.207948+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:52:13.273335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207886+0000","last_change":"2023-07-14T19:02:30.573746+0000","last_active":"2023-07-14T19:03:31.207886+0000","last_peered":"2023-07-14T19:03:31.207886+0000","last_clean":"2023-07-14T19:03:31.207886+0000","last_became_active":"2023-07-14T19:02:30.573307+0000","last_became_peered":"2023-07-14T19:02:30.573307+0000","last_unstale":"2023-07-14T19:03:31.207886+0000","last_undegraded":"2023-07-14T19:03:31.207886+0000","last_fullsized":"2023-07-14T19:03:31.207886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:49.299962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.606816+0000","last_change":"2023-07-14T19:02:30.578352+0000","last_active":"2023-07-14T19:02:35.606816+0000","last_peered":"2023-07-14T19:02:35.606816+0000","last_clean":"2023-07-14T19:02:35.606816+0000","last_became_active":"2023-07-14T19:02:30.577631+0000","last_became_peered":"2023-07-14T19:02:30.577631+0000","last_unstale":"2023-07-14T19:02:35.606816+0000","last_undegraded":"2023-07-14T19:02:35.606816+0000","last_fullsized":"2023-07-14T19:02:35.606816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:38.641986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.209130+0000","last_change":"2023-07-14T19:02:30.577813+0000","last_active":"2023-07-14T19:03:30.209130+0000","last_peered":"2023-07-14T19:03:30.209130+0000","last_clean":"2023-07-14T19:03:30.209130+0000","last_became_active":"2023-07-14T19:02:30.577482+0000","last_became_peered":"2023-07-14T19:02:30.577482+0000","last_unstale":"2023-07-14T19:03:30.209130+0000","last_undegraded":"2023-07-14T19:03:30.209130+0000","last_fullsized":"2023-07-14T19:03:30.209130+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:22:56.256630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032871+0000","last_change":"2023-07-14T19:02:30.583353+0000","last_active":"2023-07-14T19:02:36.032871+0000","last_peered":"2023-07-14T19:02:36.032871+0000","last_clean":"2023-07-14T19:02:36.032871+0000","last_became_active":"2023-07-14T19:02:30.583037+0000","last_became_peered":"2023-07-14T19:02:30.583037+0000","last_unstale":"2023-07-14T19:02:36.032871+0000","last_undegraded":"2023-07-14T19:02:36.032871+0000","last_fullsized":"2023-07-14T19:02:36.032871+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:52:50.245739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609345+0000","last_change":"2023-07-14T19:02:30.569788+0000","last_active":"2023-07-14T19:02:35.609345+0000","last_peered":"2023-07-14T19:02:35.609345+0000","last_clean":"2023-07-14T19:02:35.609345+0000","last_became_active":"2023-07-14T19:02:30.569461+0000","last_became_peered":"2023-07-14T19:02:30.569461+0000","last_unstale":"2023-07-14T19:02:35.609345+0000","last_undegraded":"2023-07-14T19:02:35.609345+0000","last_fullsized":"2023-07-14T19:02:35.609345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:35:17.252918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608767+0000","last_change":"2023-07-14T19:02:34.585220+0000","last_active":"2023-07-14T19:02:35.608767+0000","last_peered":"2023-07-14T19:02:35.608767+0000","last_clean":"2023-07-14T19:02:35.608767+0000","last_became_active":"2023-07-14T19:02:34.584771+0000","last_became_peered":"2023-07-14T19:02:34.584771+0000","last_unstale":"2023-07-14T19:02:35.608767+0000","last_undegraded":"2023-07-14T19:02:35.608767+0000","last_fullsized":"2023-07-14T19:02:35.608767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:48.195878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.208557+0000","last_change":"2023-07-14T19:02:29.938571+0000","last_active":"2023-07-14T19:02:50.208557+0000","last_peered":"2023-07-14T19:02:50.208557+0000","last_clean":"2023-07-14T19:02:50.208557+0000","last_became_active":"2023-07-14T19:02:29.938270+0000","last_became_peered":"2023-07-14T19:02:29.938270+0000","last_unstale":"2023-07-14T19:02:50.208557+0000","last_undegraded":"2023-07-14T19:02:50.208557+0000","last_fullsized":"2023-07-14T19:02:50.208557+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:24:05.121677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811982+0000","last_change":"2023-07-14T19:02:30.575894+0000","last_active":"2023-07-14T19:02:35.811982+0000","last_peered":"2023-07-14T19:02:35.811982+0000","last_clean":"2023-07-14T19:02:35.811982+0000","last_became_active":"2023-07-14T19:02:30.575607+0000","last_became_peered":"2023-07-14T19:02:30.575607+0000","last_unstale":"2023-07-14T19:02:35.811982+0000","last_undegraded":"2023-07-14T19:02:35.811982+0000","last_fullsized":"2023-07-14T19:02:35.811982+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:30.612104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218192+0000","last_change":"2023-07-14T19:03:31.218192+0000","last_active":"2023-07-14T19:03:31.218192+0000","last_peered":"2023-07-14T19:03:31.218192+0000","last_clean":"2023-07-14T19:03:31.218192+0000","last_became_active":"2023-07-14T19:03:31.217708+0000","last_became_peered":"2023-07-14T19:03:31.217708+0000","last_unstale":"2023-07-14T19:03:31.218192+0000","last_undegraded":"2023-07-14T19:03:31.218192+0000","last_fullsized":"2023-07-14T19:03:31.218192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:40.671215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.812123+0000","last_change":"2023-07-14T19:02:29.944136+0000","last_active":"2023-07-14T19:02:35.812123+0000","last_peered":"2023-07-14T19:02:35.812123+0000","last_clean":"2023-07-14T19:02:35.812123+0000","last_became_active":"2023-07-14T19:02:29.943751+0000","last_became_peered":"2023-07-14T19:02:29.943751+0000","last_unstale":"2023-07-14T19:02:35.812123+0000","last_undegraded":"2023-07-14T19:02:35.812123+0000","last_fullsized":"2023-07-14T19:02:35.812123+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:34:19.730141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208280+0000","last_change":"2023-07-14T19:02:30.583156+0000","last_active":"2023-07-14T19:03:31.208280+0000","last_peered":"2023-07-14T19:03:31.208280+0000","last_clean":"2023-07-14T19:03:31.208280+0000","last_became_active":"2023-07-14T19:02:30.582794+0000","last_became_peered":"2023-07-14T19:02:30.582794+0000","last_unstale":"2023-07-14T19:03:31.208280+0000","last_undegraded":"2023-07-14T19:03:31.208280+0000","last_fullsized":"2023-07-14T19:03:31.208280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:47:47.558145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214058+0000","last_change":"2023-07-14T19:02:34.590788+0000","last_active":"2023-07-14T19:03:31.214058+0000","last_peered":"2023-07-14T19:03:31.214058+0000","last_clean":"2023-07-14T19:03:31.214058+0000","last_became_active":"2023-07-14T19:02:34.590385+0000","last_became_peered":"2023-07-14T19:02:34.590385+0000","last_unstale":"2023-07-14T19:03:31.214058+0000","last_undegraded":"2023-07-14T19:03:31.214058+0000","last_fullsized":"2023-07-14T19:03:31.214058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:35.914846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":27,"reported_epoch":85,"state":"peering","last_fresh":"2023-07-14T19:02:50.208789+0000","last_change":"2023-07-14T19:03:30.207818+0000","last_active":"2023-07-14T19:03:30.207818+0000","last_peered":"2023-07-14T19:02:50.208789+0000","last_clean":"2023-07-14T19:02:50.208789+0000","last_became_active":"2023-07-14T19:02:29.937823+0000","last_became_peered":"2023-07-14T19:02:29.937823+0000","last_unstale":"2023-07-14T19:02:50.208789+0000","last_undegraded":"2023-07-14T19:02:50.208789+0000","last_fullsized":"2023-07-14T19:02:50.208789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607182+0000","last_change":"2023-07-14T19:02:30.582746+0000","last_active":"2023-07-14T19:02:35.607182+0000","last_peered":"2023-07-14T19:02:35.607182+0000","last_clean":"2023-07-14T19:02:35.607182+0000","last_became_active":"2023-07-14T19:02:30.582426+0000","last_became_peered":"2023-07-14T19:02:30.582426+0000","last_unstale":"2023-07-14T19:02:35.607182+0000","last_undegraded":"2023-07-14T19:02:35.607182+0000","last_fullsized":"2023-07-14T19:02:35.607182+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:14:40.875764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212948+0000","last_change":"2023-07-14T19:02:34.589966+0000","last_active":"2023-07-14T19:03:31.212948+0000","last_peered":"2023-07-14T19:03:31.212948+0000","last_clean":"2023-07-14T19:03:31.212948+0000","last_became_active":"2023-07-14T19:02:34.589544+0000","last_became_peered":"2023-07-14T19:02:34.589544+0000","last_unstale":"2023-07-14T19:03:31.212948+0000","last_undegraded":"2023-07-14T19:03:31.212948+0000","last_fullsized":"2023-07-14T19:03:31.212948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:08.813030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206624+0000","last_change":"2023-07-14T19:02:29.935507+0000","last_active":"2023-07-14T19:03:30.206624+0000","last_peered":"2023-07-14T19:03:30.206624+0000","last_clean":"2023-07-14T19:03:30.206624+0000","last_became_active":"2023-07-14T19:02:29.935160+0000","last_became_peered":"2023-07-14T19:02:29.935160+0000","last_unstale":"2023-07-14T19:03:30.206624+0000","last_undegraded":"2023-07-14T19:03:30.206624+0000","last_fullsized":"2023-07-14T19:03:30.206624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:36:48.126803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219342+0000","last_change":"2023-07-14T19:02:30.578077+0000","last_active":"2023-07-14T19:03:31.219342+0000","last_peered":"2023-07-14T19:03:31.219342+0000","last_clean":"2023-07-14T19:03:31.219342+0000","last_became_active":"2023-07-14T19:02:30.577764+0000","last_became_peered":"2023-07-14T19:02:30.577764+0000","last_unstale":"2023-07-14T19:03:31.219342+0000","last_undegraded":"2023-07-14T19:03:31.219342+0000","last_fullsized":"2023-07-14T19:03:31.219342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:40:14.188240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609074+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:02:35.609074+0000","last_peered":"2023-07-14T19:02:35.609074+0000","last_clean":"2023-07-14T19:02:35.609074+0000","last_became_active":"2023-07-14T19:02:34.583408+0000","last_became_peered":"2023-07-14T19:02:34.583408+0000","last_unstale":"2023-07-14T19:02:35.609074+0000","last_undegraded":"2023-07-14T19:02:35.609074+0000","last_fullsized":"2023-07-14T19:02:35.609074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:56:32.640321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.220191+0000","last_change":"2023-07-14T19:02:29.943899+0000","last_active":"2023-07-14T19:03:31.220191+0000","last_peered":"2023-07-14T19:03:31.220191+0000","last_clean":"2023-07-14T19:03:31.220191+0000","last_became_active":"2023-07-14T19:02:29.943585+0000","last_became_peered":"2023-07-14T19:02:29.943585+0000","last_unstale":"2023-07-14T19:03:31.220191+0000","last_undegraded":"2023-07-14T19:03:31.220191+0000","last_fullsized":"2023-07-14T19:03:31.220191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:29.940165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.207480+0000","last_change":"2023-07-14T19:02:30.567147+0000","last_active":"2023-07-14T19:03:30.207480+0000","last_peered":"2023-07-14T19:03:30.207480+0000","last_clean":"2023-07-14T19:03:30.207480+0000","last_became_active":"2023-07-14T19:02:30.566792+0000","last_became_peered":"2023-07-14T19:02:30.566792+0000","last_unstale":"2023-07-14T19:03:30.207480+0000","last_undegraded":"2023-07-14T19:03:30.207480+0000","last_fullsized":"2023-07-14T19:03:30.207480+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:56.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609090+0000","last_change":"2023-07-14T19:02:34.579561+0000","last_active":"2023-07-14T19:02:35.609090+0000","last_peered":"2023-07-14T19:02:35.609090+0000","last_clean":"2023-07-14T19:02:35.609090+0000","last_became_active":"2023-07-14T19:02:34.579099+0000","last_became_peered":"2023-07-14T19:02:34.579099+0000","last_unstale":"2023-07-14T19:02:35.609090+0000","last_undegraded":"2023-07-14T19:02:35.609090+0000","last_fullsized":"2023-07-14T19:02:35.609090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:24:00.439199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.209801+0000","last_change":"2023-07-14T19:02:29.936651+0000","last_active":"2023-07-14T19:02:50.209801+0000","last_peered":"2023-07-14T19:02:50.209801+0000","last_clean":"2023-07-14T19:02:50.209801+0000","last_became_active":"2023-07-14T19:02:29.936280+0000","last_became_peered":"2023-07-14T19:02:29.936280+0000","last_unstale":"2023-07-14T19:02:50.209801+0000","last_undegraded":"2023-07-14T19:02:50.209801+0000","last_fullsized":"2023-07-14T19:02:50.209801+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:28.735813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.220085+0000","last_change":"2023-07-14T19:02:30.575925+0000","last_active":"2023-07-14T19:03:31.220085+0000","last_peered":"2023-07-14T19:03:31.220085+0000","last_clean":"2023-07-14T19:03:31.220085+0000","last_became_active":"2023-07-14T19:02:30.575583+0000","last_became_peered":"2023-07-14T19:02:30.575583+0000","last_unstale":"2023-07-14T19:03:31.220085+0000","last_undegraded":"2023-07-14T19:03:31.220085+0000","last_fullsized":"2023-07-14T19:03:31.220085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:11:02.245125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208976+0000","last_change":"2023-07-14T19:02:34.587094+0000","last_active":"2023-07-14T19:03:30.208976+0000","last_peered":"2023-07-14T19:03:30.208976+0000","last_clean":"2023-07-14T19:03:30.208976+0000","last_became_active":"2023-07-14T19:02:34.586650+0000","last_became_peered":"2023-07-14T19:02:34.586650+0000","last_unstale":"2023-07-14T19:03:30.208976+0000","last_undegraded":"2023-07-14T19:03:30.208976+0000","last_fullsized":"2023-07-14T19:03:30.208976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:59:14.503188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.209484+0000","last_change":"2023-07-14T19:02:29.934651+0000","last_active":"2023-07-14T19:02:50.209484+0000","last_peered":"2023-07-14T19:02:50.209484+0000","last_clean":"2023-07-14T19:02:50.209484+0000","last_became_active":"2023-07-14T19:02:29.934347+0000","last_became_peered":"2023-07-14T19:02:29.934347+0000","last_unstale":"2023-07-14T19:02:50.209484+0000","last_undegraded":"2023-07-14T19:02:50.209484+0000","last_fullsized":"2023-07-14T19:02:50.209484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:34:23.476531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811237+0000","last_change":"2023-07-14T19:02:30.571680+0000","last_active":"2023-07-14T19:02:35.811237+0000","last_peered":"2023-07-14T19:02:35.811237+0000","last_clean":"2023-07-14T19:02:35.811237+0000","last_became_active":"2023-07-14T19:02:30.571367+0000","last_became_peered":"2023-07-14T19:02:30.571367+0000","last_unstale":"2023-07-14T19:02:35.811237+0000","last_undegraded":"2023-07-14T19:02:35.811237+0000","last_fullsized":"2023-07-14T19:02:35.811237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:43.958053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208272+0000","last_change":"2023-07-14T19:02:34.577642+0000","last_active":"2023-07-14T19:03:31.208272+0000","last_peered":"2023-07-14T19:03:31.208272+0000","last_clean":"2023-07-14T19:03:31.208272+0000","last_became_active":"2023-07-14T19:02:34.577232+0000","last_became_peered":"2023-07-14T19:02:34.577232+0000","last_unstale":"2023-07-14T19:03:31.208272+0000","last_undegraded":"2023-07-14T19:03:31.208272+0000","last_fullsized":"2023-07-14T19:03:31.208272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:19.952531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218302+0000","last_change":"2023-07-14T19:02:29.940688+0000","last_active":"2023-07-14T19:03:31.218302+0000","last_peered":"2023-07-14T19:03:31.218302+0000","last_clean":"2023-07-14T19:03:31.218302+0000","last_became_active":"2023-07-14T19:02:29.940444+0000","last_became_peered":"2023-07-14T19:02:29.940444+0000","last_unstale":"2023-07-14T19:03:31.218302+0000","last_undegraded":"2023-07-14T19:03:31.218302+0000","last_fullsized":"2023-07-14T19:03:31.218302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:37.769743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032080+0000","last_change":"2023-07-14T19:02:30.568127+0000","last_active":"2023-07-14T19:02:36.032080+0000","last_peered":"2023-07-14T19:02:36.032080+0000","last_clean":"2023-07-14T19:02:36.032080+0000","last_became_active":"2023-07-14T19:02:30.567811+0000","last_became_peered":"2023-07-14T19:02:30.567811+0000","last_unstale":"2023-07-14T19:02:36.032080+0000","last_undegraded":"2023-07-14T19:02:36.032080+0000","last_fullsized":"2023-07-14T19:02:36.032080+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:07:11.513097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811843+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:02:35.811843+0000","last_peered":"2023-07-14T19:02:35.811843+0000","last_clean":"2023-07-14T19:02:35.811843+0000","last_became_active":"2023-07-14T19:02:34.583480+0000","last_became_peered":"2023-07-14T19:02:34.583480+0000","last_unstale":"2023-07-14T19:02:35.811843+0000","last_undegraded":"2023-07-14T19:02:35.811843+0000","last_fullsized":"2023-07-14T19:02:35.811843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:27.737212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":63,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:03:26.670349+0000","last_change":"2023-07-14T19:02:29.938459+0000","last_active":"2023-07-14T19:03:26.670349+0000","last_peered":"2023-07-14T19:03:26.670349+0000","last_clean":"2023-07-14T19:03:26.670349+0000","last_became_active":"2023-07-14T19:02:29.938212+0000","last_became_peered":"2023-07-14T19:02:29.938212+0000","last_unstale":"2023-07-14T19:03:26.670349+0000","last_undegraded":"2023-07-14T19:03:26.670349+0000","last_fullsized":"2023-07-14T19:03:26.670349+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:52:39.556855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":38,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607453+0000","last_change":"2023-07-14T19:02:30.586345+0000","last_active":"2023-07-14T19:02:35.607453+0000","last_peered":"2023-07-14T19:02:35.607453+0000","last_clean":"2023-07-14T19:02:35.607453+0000","last_became_active":"2023-07-14T19:02:30.585949+0000","last_became_peered":"2023-07-14T19:02:30.585949+0000","last_unstale":"2023-07-14T19:02:35.607453+0000","last_undegraded":"2023-07-14T19:02:35.607453+0000","last_fullsized":"2023-07-14T19:02:35.607453+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:34:26.531581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.031989+0000","last_change":"2023-07-14T19:02:34.582922+0000","last_active":"2023-07-14T19:02:36.031989+0000","last_peered":"2023-07-14T19:02:36.031989+0000","last_clean":"2023-07-14T19:02:36.031989+0000","last_became_active":"2023-07-14T19:02:34.582146+0000","last_became_peered":"2023-07-14T19:02:34.582146+0000","last_unstale":"2023-07-14T19:02:36.031989+0000","last_undegraded":"2023-07-14T19:02:36.031989+0000","last_fullsized":"2023-07-14T19:02:36.031989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:10:17.978844+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214601+0000","last_change":"2023-07-14T19:02:29.938215+0000","last_active":"2023-07-14T19:03:30.214601+0000","last_peered":"2023-07-14T19:03:30.214601+0000","last_clean":"2023-07-14T19:03:30.214601+0000","last_became_active":"2023-07-14T19:02:29.937889+0000","last_became_peered":"2023-07-14T19:02:29.937889+0000","last_unstale":"2023-07-14T19:03:30.214601+0000","last_undegraded":"2023-07-14T19:03:30.214601+0000","last_fullsized":"2023-07-14T19:03:30.214601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:51.892537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208284+0000","last_change":"2023-07-14T19:02:30.569622+0000","last_active":"2023-07-14T19:03:31.208284+0000","last_peered":"2023-07-14T19:03:31.208284+0000","last_clean":"2023-07-14T19:03:31.208284+0000","last_became_active":"2023-07-14T19:02:30.569201+0000","last_became_peered":"2023-07-14T19:02:30.569201+0000","last_unstale":"2023-07-14T19:03:31.208284+0000","last_undegraded":"2023-07-14T19:03:31.208284+0000","last_fullsized":"2023-07-14T19:03:31.208284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:53:07.451914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214242+0000","last_change":"2023-07-14T19:02:34.580978+0000","last_active":"2023-07-14T19:03:31.214242+0000","last_peered":"2023-07-14T19:03:31.214242+0000","last_clean":"2023-07-14T19:03:31.214242+0000","last_became_active":"2023-07-14T19:02:34.580503+0000","last_became_peered":"2023-07-14T19:02:34.580503+0000","last_unstale":"2023-07-14T19:03:31.214242+0000","last_undegraded":"2023-07-14T19:03:31.214242+0000","last_fullsized":"2023-07-14T19:03:31.214242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:06.819824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219409+0000","last_change":"2023-07-14T19:02:29.942400+0000","last_active":"2023-07-14T19:03:31.219409+0000","last_peered":"2023-07-14T19:03:31.219409+0000","last_clean":"2023-07-14T19:03:31.219409+0000","last_became_active":"2023-07-14T19:02:29.942080+0000","last_became_peered":"2023-07-14T19:02:29.942080+0000","last_unstale":"2023-07-14T19:03:31.219409+0000","last_undegraded":"2023-07-14T19:03:31.219409+0000","last_fullsized":"2023-07-14T19:03:31.219409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:09:06.130256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607297+0000","last_change":"2023-07-14T19:02:30.582869+0000","last_active":"2023-07-14T19:02:35.607297+0000","last_peered":"2023-07-14T19:02:35.607297+0000","last_clean":"2023-07-14T19:02:35.607297+0000","last_became_active":"2023-07-14T19:02:30.582558+0000","last_became_peered":"2023-07-14T19:02:30.582558+0000","last_unstale":"2023-07-14T19:02:35.607297+0000","last_undegraded":"2023-07-14T19:02:35.607297+0000","last_fullsized":"2023-07-14T19:02:35.607297+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:56:31.547355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033051+0000","last_change":"2023-07-14T19:02:34.582610+0000","last_active":"2023-07-14T19:02:36.033051+0000","last_peered":"2023-07-14T19:02:36.033051+0000","last_clean":"2023-07-14T19:02:36.033051+0000","last_became_active":"2023-07-14T19:02:34.581961+0000","last_became_peered":"2023-07-14T19:02:34.581961+0000","last_unstale":"2023-07-14T19:02:36.033051+0000","last_undegraded":"2023-07-14T19:02:36.033051+0000","last_fullsized":"2023-07-14T19:02:36.033051+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:28:06.554521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'30","reported_seq":135,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:03:27.679842+0000","last_change":"2023-07-14T19:02:29.939314+0000","last_active":"2023-07-14T19:03:27.679842+0000","last_peered":"2023-07-14T19:03:27.679842+0000","last_clean":"2023-07-14T19:03:27.679842+0000","last_became_active":"2023-07-14T19:02:29.939000+0000","last_became_peered":"2023-07-14T19:02:29.939000+0000","last_unstale":"2023-07-14T19:03:27.679842+0000","last_undegraded":"2023-07-14T19:03:27.679842+0000","last_fullsized":"2023-07-14T19:03:27.679842+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:44:31.404753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56970,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":83,"num_read_kb":97,"num_write":31,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206481+0000","last_change":"2023-07-14T19:02:30.572052+0000","last_active":"2023-07-14T19:03:30.206481+0000","last_peered":"2023-07-14T19:03:30.206481+0000","last_clean":"2023-07-14T19:03:30.206481+0000","last_became_active":"2023-07-14T19:02:30.571739+0000","last_became_peered":"2023-07-14T19:02:30.571739+0000","last_unstale":"2023-07-14T19:03:30.206481+0000","last_undegraded":"2023-07-14T19:03:30.206481+0000","last_fullsized":"2023-07-14T19:03:30.206481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:03.443418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219888+0000","last_change":"2023-07-14T19:02:34.576663+0000","last_active":"2023-07-14T19:03:31.219888+0000","last_peered":"2023-07-14T19:03:31.219888+0000","last_clean":"2023-07-14T19:03:31.219888+0000","last_became_active":"2023-07-14T19:02:34.576162+0000","last_became_peered":"2023-07-14T19:02:34.576162+0000","last_unstale":"2023-07-14T19:03:31.219888+0000","last_undegraded":"2023-07-14T19:03:31.219888+0000","last_fullsized":"2023-07-14T19:03:31.219888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:50.099226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":2,"reported_epoch":86,"state":"peering","last_fresh":"2023-07-14T19:02:50.616451+0000","last_change":"2023-07-14T19:03:30.725628+0000","last_active":"2023-07-14T19:03:30.725628+0000","last_peered":"2023-07-14T19:02:50.616451+0000","last_clean":"2023-07-14T19:02:50.616451+0000","last_became_active":"2023-07-14T19:02:29.938630+0000","last_became_peered":"2023-07-14T19:02:29.938630+0000","last_unstale":"2023-07-14T19:02:50.616451+0000","last_undegraded":"2023-07-14T19:02:50.616451+0000","last_fullsized":"2023-07-14T19:02:50.616451+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608493+0000","last_change":"2023-07-14T19:02:30.573798+0000","last_active":"2023-07-14T19:02:35.608493+0000","last_peered":"2023-07-14T19:02:35.608493+0000","last_clean":"2023-07-14T19:02:35.608493+0000","last_became_active":"2023-07-14T19:02:30.572568+0000","last_became_peered":"2023-07-14T19:02:30.572568+0000","last_unstale":"2023-07-14T19:02:35.608493+0000","last_undegraded":"2023-07-14T19:02:35.608493+0000","last_fullsized":"2023-07-14T19:02:35.608493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:02.901707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208768+0000","last_change":"2023-07-14T19:02:34.582256+0000","last_active":"2023-07-14T19:03:30.208768+0000","last_peered":"2023-07-14T19:03:30.208768+0000","last_clean":"2023-07-14T19:03:30.208768+0000","last_became_active":"2023-07-14T19:02:34.581799+0000","last_became_peered":"2023-07-14T19:02:34.581799+0000","last_unstale":"2023-07-14T19:03:30.208768+0000","last_undegraded":"2023-07-14T19:03:30.208768+0000","last_fullsized":"2023-07-14T19:03:30.208768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:12.667561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219545+0000","last_change":"2023-07-14T19:02:29.943874+0000","last_active":"2023-07-14T19:03:31.219545+0000","last_peered":"2023-07-14T19:03:31.219545+0000","last_clean":"2023-07-14T19:03:31.219545+0000","last_became_active":"2023-07-14T19:02:29.943596+0000","last_became_peered":"2023-07-14T19:02:29.943596+0000","last_unstale":"2023-07-14T19:03:31.219545+0000","last_undegraded":"2023-07-14T19:03:31.219545+0000","last_fullsized":"2023-07-14T19:03:31.219545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:17.878050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214284+0000","last_change":"2023-07-14T19:02:30.572157+0000","last_active":"2023-07-14T19:03:30.214284+0000","last_peered":"2023-07-14T19:03:30.214284+0000","last_clean":"2023-07-14T19:03:30.214284+0000","last_became_active":"2023-07-14T19:02:30.571877+0000","last_became_peered":"2023-07-14T19:02:30.571877+0000","last_unstale":"2023-07-14T19:03:30.214284+0000","last_undegraded":"2023-07-14T19:03:30.214284+0000","last_fullsized":"2023-07-14T19:03:30.214284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:15:54.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.031996+0000","last_change":"2023-07-14T19:02:34.588594+0000","last_active":"2023-07-14T19:02:36.031996+0000","last_peered":"2023-07-14T19:02:36.031996+0000","last_clean":"2023-07-14T19:02:36.031996+0000","last_became_active":"2023-07-14T19:02:34.587763+0000","last_became_peered":"2023-07-14T19:02:34.587763+0000","last_unstale":"2023-07-14T19:02:36.031996+0000","last_undegraded":"2023-07-14T19:02:36.031996+0000","last_fullsized":"2023-07-14T19:02:36.031996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:13:45.839894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212239+0000","last_change":"2023-07-14T19:02:29.939145+0000","last_active":"2023-07-14T19:03:31.212239+0000","last_peered":"2023-07-14T19:03:31.212239+0000","last_clean":"2023-07-14T19:03:31.212239+0000","last_became_active":"2023-07-14T19:02:29.938816+0000","last_became_peered":"2023-07-14T19:02:29.938816+0000","last_unstale":"2023-07-14T19:03:31.212239+0000","last_undegraded":"2023-07-14T19:03:31.212239+0000","last_fullsized":"2023-07-14T19:03:31.212239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:05.245627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.213928+0000","last_change":"2023-07-14T19:02:30.572694+0000","last_active":"2023-07-14T19:03:30.213928+0000","last_peered":"2023-07-14T19:03:30.213928+0000","last_clean":"2023-07-14T19:03:30.213928+0000","last_became_active":"2023-07-14T19:02:30.572391+0000","last_became_peered":"2023-07-14T19:02:30.572391+0000","last_unstale":"2023-07-14T19:03:30.213928+0000","last_undegraded":"2023-07-14T19:03:30.213928+0000","last_fullsized":"2023-07-14T19:03:30.213928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:21.348702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609130+0000","last_change":"2023-07-14T19:02:34.582476+0000","last_active":"2023-07-14T19:02:35.609130+0000","last_peered":"2023-07-14T19:02:35.609130+0000","last_clean":"2023-07-14T19:02:35.609130+0000","last_became_active":"2023-07-14T19:02:34.582083+0000","last_became_peered":"2023-07-14T19:02:34.582083+0000","last_unstale":"2023-07-14T19:02:35.609130+0000","last_undegraded":"2023-07-14T19:02:35.609130+0000","last_fullsized":"2023-07-14T19:02:35.609130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:53.732894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":2,"reported_epoch":86,"state":"peering","last_fresh":"2023-07-14T19:02:49.197385+0000","last_change":"2023-07-14T19:03:30.725597+0000","last_active":"2023-07-14T19:03:30.725597+0000","last_peered":"2023-07-14T19:02:49.197385+0000","last_clean":"2023-07-14T19:02:49.197385+0000","last_became_active":"2023-07-14T19:02:29.941103+0000","last_became_peered":"2023-07-14T19:02:29.941103+0000","last_unstale":"2023-07-14T19:02:49.197385+0000","last_undegraded":"2023-07-14T19:02:49.197385+0000","last_fullsized":"2023-07-14T19:02:49.197385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.282151+0000","last_change":"2023-07-14T19:03:31.282151+0000","last_active":"2023-07-14T19:03:31.282151+0000","last_peered":"2023-07-14T19:03:31.282151+0000","last_clean":"2023-07-14T19:03:31.282151+0000","last_became_active":"2023-07-14T19:03:31.281763+0000","last_became_peered":"2023-07-14T19:03:31.281763+0000","last_unstale":"2023-07-14T19:03:31.282151+0000","last_undegraded":"2023-07-14T19:03:31.282151+0000","last_fullsized":"2023-07-14T19:03:31.282151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:13.429632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218880+0000","last_change":"2023-07-14T19:02:34.584170+0000","last_active":"2023-07-14T19:03:31.218880+0000","last_peered":"2023-07-14T19:03:31.218880+0000","last_clean":"2023-07-14T19:03:31.218880+0000","last_became_active":"2023-07-14T19:02:34.583773+0000","last_became_peered":"2023-07-14T19:02:34.583773+0000","last_unstale":"2023-07-14T19:03:31.218880+0000","last_undegraded":"2023-07-14T19:03:31.218880+0000","last_fullsized":"2023-07-14T19:03:31.218880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:32:31.049038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.207509+0000","last_change":"2023-07-14T19:02:29.938386+0000","last_active":"2023-07-14T19:03:30.207509+0000","last_peered":"2023-07-14T19:03:30.207509+0000","last_clean":"2023-07-14T19:03:30.207509+0000","last_became_active":"2023-07-14T19:02:29.937394+0000","last_became_peered":"2023-07-14T19:02:29.937394+0000","last_unstale":"2023-07-14T19:03:30.207509+0000","last_undegraded":"2023-07-14T19:03:30.207509+0000","last_fullsized":"2023-07-14T19:03:30.207509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:58.170429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208774+0000","last_change":"2023-07-14T19:02:30.572485+0000","last_active":"2023-07-14T19:03:30.208774+0000","last_peered":"2023-07-14T19:03:30.208774+0000","last_clean":"2023-07-14T19:03:30.208774+0000","last_became_active":"2023-07-14T19:02:30.572177+0000","last_became_peered":"2023-07-14T19:02:30.572177+0000","last_unstale":"2023-07-14T19:03:30.208774+0000","last_undegraded":"2023-07-14T19:03:30.208774+0000","last_fullsized":"2023-07-14T19:03:30.208774+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:56:12.967413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.207256+0000","last_change":"2023-07-14T19:02:34.582340+0000","last_active":"2023-07-14T19:03:30.207256+0000","last_peered":"2023-07-14T19:03:30.207256+0000","last_clean":"2023-07-14T19:03:30.207256+0000","last_became_active":"2023-07-14T19:02:34.581936+0000","last_became_peered":"2023-07-14T19:02:34.581936+0000","last_unstale":"2023-07-14T19:03:30.207256+0000","last_undegraded":"2023-07-14T19:03:30.207256+0000","last_fullsized":"2023-07-14T19:03:30.207256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:31.710972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032203+0000","last_change":"2023-07-14T19:02:29.941404+0000","last_active":"2023-07-14T19:02:36.032203+0000","last_peered":"2023-07-14T19:02:36.032203+0000","last_clean":"2023-07-14T19:02:36.032203+0000","last_became_active":"2023-07-14T19:02:29.941052+0000","last_became_peered":"2023-07-14T19:02:29.941052+0000","last_unstale":"2023-07-14T19:02:36.032203+0000","last_undegraded":"2023-07-14T19:02:36.032203+0000","last_fullsized":"2023-07-14T19:02:36.032203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:05:50.738003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033689+0000","last_change":"2023-07-14T19:02:30.565671+0000","last_active":"2023-07-14T19:02:36.033689+0000","last_peered":"2023-07-14T19:02:36.033689+0000","last_clean":"2023-07-14T19:02:36.033689+0000","last_became_active":"2023-07-14T19:02:30.565375+0000","last_became_peered":"2023-07-14T19:02:30.565375+0000","last_unstale":"2023-07-14T19:02:36.033689+0000","last_undegraded":"2023-07-14T19:02:36.033689+0000","last_fullsized":"2023-07-14T19:02:36.033689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:00.244922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608815+0000","last_change":"2023-07-14T19:02:34.577692+0000","last_active":"2023-07-14T19:02:35.608815+0000","last_peered":"2023-07-14T19:02:35.608815+0000","last_clean":"2023-07-14T19:02:35.608815+0000","last_became_active":"2023-07-14T19:02:34.577329+0000","last_became_peered":"2023-07-14T19:02:34.577329+0000","last_unstale":"2023-07-14T19:02:35.608815+0000","last_undegraded":"2023-07-14T19:02:35.608815+0000","last_fullsized":"2023-07-14T19:02:35.608815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:42.179550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214219+0000","last_change":"2023-07-14T19:02:29.938330+0000","last_active":"2023-07-14T19:03:30.214219+0000","last_peered":"2023-07-14T19:03:30.214219+0000","last_clean":"2023-07-14T19:03:30.214219+0000","last_became_active":"2023-07-14T19:02:29.938022+0000","last_became_peered":"2023-07-14T19:02:29.938022+0000","last_unstale":"2023-07-14T19:03:30.214219+0000","last_undegraded":"2023-07-14T19:03:30.214219+0000","last_fullsized":"2023-07-14T19:03:30.214219+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:53.432786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.033320+0000","last_change":"2023-07-14T19:02:30.569353+0000","last_active":"2023-07-14T19:02:36.033320+0000","last_peered":"2023-07-14T19:02:36.033320+0000","last_clean":"2023-07-14T19:02:36.033320+0000","last_became_active":"2023-07-14T19:02:30.569037+0000","last_became_peered":"2023-07-14T19:02:30.569037+0000","last_unstale":"2023-07-14T19:02:36.033320+0000","last_undegraded":"2023-07-14T19:02:36.033320+0000","last_fullsized":"2023-07-14T19:02:36.033320+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:43.690826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609120+0000","last_change":"2023-07-14T19:02:34.580786+0000","last_active":"2023-07-14T19:02:35.609120+0000","last_peered":"2023-07-14T19:02:35.609120+0000","last_clean":"2023-07-14T19:02:35.609120+0000","last_became_active":"2023-07-14T19:02:34.580395+0000","last_became_peered":"2023-07-14T19:02:34.580395+0000","last_unstale":"2023-07-14T19:02:35.609120+0000","last_undegraded":"2023-07-14T19:02:35.609120+0000","last_fullsized":"2023-07-14T19:02:35.609120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:24.810182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218850+0000","last_change":"2023-07-14T19:02:30.574302+0000","last_active":"2023-07-14T19:03:31.218850+0000","last_peered":"2023-07-14T19:03:31.218850+0000","last_clean":"2023-07-14T19:03:31.218850+0000","last_became_active":"2023-07-14T19:02:30.573912+0000","last_became_peered":"2023-07-14T19:02:30.573912+0000","last_unstale":"2023-07-14T19:03:31.218850+0000","last_undegraded":"2023-07-14T19:03:31.218850+0000","last_fullsized":"2023-07-14T19:03:31.218850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:57.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:03:28.673439+0000","last_change":"2023-07-14T19:02:29.942302+0000","last_active":"2023-07-14T19:03:28.673439+0000","last_peered":"2023-07-14T19:03:28.673439+0000","last_clean":"2023-07-14T19:03:28.673439+0000","last_became_active":"2023-07-14T19:02:29.942005+0000","last_became_peered":"2023-07-14T19:02:29.942005+0000","last_unstale":"2023-07-14T19:03:28.673439+0000","last_undegraded":"2023-07-14T19:03:28.673439+0000","last_fullsized":"2023-07-14T19:03:28.673439+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:55.061870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214474+0000","last_change":"2023-07-14T19:02:34.578248+0000","last_active":"2023-07-14T19:03:30.214474+0000","last_peered":"2023-07-14T19:03:30.214474+0000","last_clean":"2023-07-14T19:03:30.214474+0000","last_became_active":"2023-07-14T19:02:34.577930+0000","last_became_peered":"2023-07-14T19:02:34.577930+0000","last_unstale":"2023-07-14T19:03:30.214474+0000","last_undegraded":"2023-07-14T19:03:30.214474+0000","last_fullsized":"2023-07-14T19:03:30.214474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:40:19.412781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811402+0000","last_change":"2023-07-14T19:02:30.582892+0000","last_active":"2023-07-14T19:02:35.811402+0000","last_peered":"2023-07-14T19:02:35.811402+0000","last_clean":"2023-07-14T19:02:35.811402+0000","last_became_active":"2023-07-14T19:02:30.582588+0000","last_became_peered":"2023-07-14T19:02:30.582588+0000","last_unstale":"2023-07-14T19:02:35.811402+0000","last_undegraded":"2023-07-14T19:02:35.811402+0000","last_fullsized":"2023-07-14T19:02:35.811402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:00:14.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214203+0000","last_change":"2023-07-14T19:02:29.935857+0000","last_active":"2023-07-14T19:03:31.214203+0000","last_peered":"2023-07-14T19:03:31.214203+0000","last_clean":"2023-07-14T19:03:31.214203+0000","last_became_active":"2023-07-14T19:02:29.935489+0000","last_became_peered":"2023-07-14T19:02:29.935489+0000","last_unstale":"2023-07-14T19:03:31.214203+0000","last_undegraded":"2023-07-14T19:03:31.214203+0000","last_fullsized":"2023-07-14T19:03:31.214203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:08.956235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":338,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.213796+0000","last_change":"2023-07-14T19:01:31.967476+0000","last_active":"2023-07-14T19:03:30.213796+0000","last_peered":"2023-07-14T19:03:30.213796+0000","last_clean":"2023-07-14T19:03:30.213796+0000","last_became_active":"2023-07-14T19:01:31.967146+0000","last_became_peered":"2023-07-14T19:01:31.967146+0000","last_unstale":"2023-07-14T19:03:30.213796+0000","last_undegraded":"2023-07-14T19:03:30.213796+0000","last_fullsized":"2023-07-14T19:03:30.213796+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_clean_scrub_stamp":"2023-07-14T18:58:40.634620+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:56.581484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219439+0000","last_change":"2023-07-14T19:02:34.571823+0000","last_active":"2023-07-14T19:03:31.219439+0000","last_peered":"2023-07-14T19:03:31.219439+0000","last_clean":"2023-07-14T19:03:31.219439+0000","last_became_active":"2023-07-14T19:02:34.571359+0000","last_became_peered":"2023-07-14T19:02:34.571359+0000","last_unstale":"2023-07-14T19:03:31.219439+0000","last_undegraded":"2023-07-14T19:03:31.219439+0000","last_fullsized":"2023-07-14T19:03:31.219439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:28.423706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608304+0000","last_change":"2023-07-14T19:02:30.573736+0000","last_active":"2023-07-14T19:02:35.608304+0000","last_peered":"2023-07-14T19:02:35.608304+0000","last_clean":"2023-07-14T19:02:35.608304+0000","last_became_active":"2023-07-14T19:02:30.573381+0000","last_became_peered":"2023-07-14T19:02:30.573381+0000","last_unstale":"2023-07-14T19:02:35.608304+0000","last_undegraded":"2023-07-14T19:02:35.608304+0000","last_fullsized":"2023-07-14T19:02:35.608304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:21:36.599123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.213908+0000","last_change":"2023-07-14T19:02:29.937660+0000","last_active":"2023-07-14T19:03:30.213908+0000","last_peered":"2023-07-14T19:03:30.213908+0000","last_clean":"2023-07-14T19:03:30.213908+0000","last_became_active":"2023-07-14T19:02:29.937348+0000","last_became_peered":"2023-07-14T19:02:29.937348+0000","last_unstale":"2023-07-14T19:03:30.213908+0000","last_undegraded":"2023-07-14T19:03:30.213908+0000","last_fullsized":"2023-07-14T19:03:30.213908+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:11:25.564711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.209687+0000","last_change":"2023-07-14T19:02:34.581534+0000","last_active":"2023-07-14T19:03:30.209687+0000","last_peered":"2023-07-14T19:03:30.209687+0000","last_clean":"2023-07-14T19:03:30.209687+0000","last_became_active":"2023-07-14T19:02:34.580998+0000","last_became_peered":"2023-07-14T19:02:34.580998+0000","last_unstale":"2023-07-14T19:03:30.209687+0000","last_undegraded":"2023-07-14T19:03:30.209687+0000","last_fullsized":"2023-07-14T19:03:30.209687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:51.597227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206926+0000","last_change":"2023-07-14T19:02:29.942336+0000","last_active":"2023-07-14T19:03:30.206926+0000","last_peered":"2023-07-14T19:03:30.206926+0000","last_clean":"2023-07-14T19:03:30.206926+0000","last_became_active":"2023-07-14T19:02:29.942033+0000","last_became_peered":"2023-07-14T19:02:29.942033+0000","last_unstale":"2023-07-14T19:03:30.206926+0000","last_undegraded":"2023-07-14T19:03:30.206926+0000","last_fullsized":"2023-07-14T19:03:30.206926+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:32.682171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608984+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:02:35.608984+0000","last_peered":"2023-07-14T19:02:35.608984+0000","last_clean":"2023-07-14T19:02:35.608984+0000","last_became_active":"2023-07-14T19:02:30.571129+0000","last_became_peered":"2023-07-14T19:02:30.571129+0000","last_unstale":"2023-07-14T19:02:35.608984+0000","last_undegraded":"2023-07-14T19:02:35.608984+0000","last_fullsized":"2023-07-14T19:02:35.608984+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:17.668104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811239+0000","last_change":"2023-07-14T19:02:34.580740+0000","last_active":"2023-07-14T19:02:35.811239+0000","last_peered":"2023-07-14T19:02:35.811239+0000","last_clean":"2023-07-14T19:02:35.811239+0000","last_became_active":"2023-07-14T19:02:34.580344+0000","last_became_peered":"2023-07-14T19:02:34.580344+0000","last_unstale":"2023-07-14T19:02:35.811239+0000","last_undegraded":"2023-07-14T19:02:35.811239+0000","last_fullsized":"2023-07-14T19:02:35.811239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:26:55.627423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'5","reported_seq":27,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:54.921152+0000","last_change":"2023-07-14T19:02:29.941349+0000","last_active":"2023-07-14T19:02:54.921152+0000","last_peered":"2023-07-14T19:02:54.921152+0000","last_clean":"2023-07-14T19:02:54.921152+0000","last_became_active":"2023-07-14T19:02:29.941039+0000","last_became_peered":"2023-07-14T19:02:29.941039+0000","last_unstale":"2023-07-14T19:02:54.921152+0000","last_undegraded":"2023-07-14T19:02:54.921152+0000","last_fullsized":"2023-07-14T19:02:54.921152+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:47:27.500876+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811807+0000","last_change":"2023-07-14T19:02:30.577546+0000","last_active":"2023-07-14T19:02:35.811807+0000","last_peered":"2023-07-14T19:02:35.811807+0000","last_clean":"2023-07-14T19:02:35.811807+0000","last_became_active":"2023-07-14T19:02:30.576833+0000","last_became_peered":"2023-07-14T19:02:30.576833+0000","last_unstale":"2023-07-14T19:02:35.811807+0000","last_undegraded":"2023-07-14T19:02:35.811807+0000","last_fullsized":"2023-07-14T19:02:35.811807+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:47.060524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608552+0000","last_change":"2023-07-14T19:02:34.580655+0000","last_active":"2023-07-14T19:02:35.608552+0000","last_peered":"2023-07-14T19:02:35.608552+0000","last_clean":"2023-07-14T19:02:35.608552+0000","last_became_active":"2023-07-14T19:02:34.580186+0000","last_became_peered":"2023-07-14T19:02:34.580186+0000","last_unstale":"2023-07-14T19:02:35.608552+0000","last_undegraded":"2023-07-14T19:02:35.608552+0000","last_fullsized":"2023-07-14T19:02:35.608552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:30.211525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212620+0000","last_change":"2023-07-14T19:02:29.938343+0000","last_active":"2023-07-14T19:03:31.212620+0000","last_peered":"2023-07-14T19:03:31.212620+0000","last_clean":"2023-07-14T19:03:31.212620+0000","last_became_active":"2023-07-14T19:02:29.938053+0000","last_became_peered":"2023-07-14T19:02:29.938053+0000","last_unstale":"2023-07-14T19:03:31.212620+0000","last_undegraded":"2023-07-14T19:03:31.212620+0000","last_fullsized":"2023-07-14T19:03:31.212620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:50:45.655040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208419+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:03:31.208419+0000","last_peered":"2023-07-14T19:03:31.208419+0000","last_clean":"2023-07-14T19:03:31.208419+0000","last_became_active":"2023-07-14T19:02:30.572436+0000","last_became_peered":"2023-07-14T19:02:30.572436+0000","last_unstale":"2023-07-14T19:03:31.208419+0000","last_undegraded":"2023-07-14T19:03:31.208419+0000","last_fullsized":"2023-07-14T19:03:31.208419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:11:10.005214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213237+0000","last_change":"2023-07-14T19:02:34.582900+0000","last_active":"2023-07-14T19:03:31.213237+0000","last_peered":"2023-07-14T19:03:31.213237+0000","last_clean":"2023-07-14T19:03:31.213237+0000","last_became_active":"2023-07-14T19:02:34.582493+0000","last_became_peered":"2023-07-14T19:02:34.582493+0000","last_unstale":"2023-07-14T19:03:31.213237+0000","last_undegraded":"2023-07-14T19:03:31.213237+0000","last_fullsized":"2023-07-14T19:03:31.213237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:00:45.030002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.199746+0000","last_change":"2023-07-14T19:02:29.941819+0000","last_active":"2023-07-14T19:02:49.199746+0000","last_peered":"2023-07-14T19:02:49.199746+0000","last_clean":"2023-07-14T19:02:49.199746+0000","last_became_active":"2023-07-14T19:02:29.941469+0000","last_became_peered":"2023-07-14T19:02:29.941469+0000","last_unstale":"2023-07-14T19:02:49.199746+0000","last_undegraded":"2023-07-14T19:02:49.199746+0000","last_fullsized":"2023-07-14T19:02:49.199746+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:34:27.032813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206177+0000","last_change":"2023-07-14T19:02:30.565125+0000","last_active":"2023-07-14T19:03:30.206177+0000","last_peered":"2023-07-14T19:03:30.206177+0000","last_clean":"2023-07-14T19:03:30.206177+0000","last_became_active":"2023-07-14T19:02:30.564820+0000","last_became_peered":"2023-07-14T19:02:30.564820+0000","last_unstale":"2023-07-14T19:03:30.206177+0000","last_undegraded":"2023-07-14T19:03:30.206177+0000","last_fullsized":"2023-07-14T19:03:30.206177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:54:35.924491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:36.032950+0000","last_change":"2023-07-14T19:02:34.588273+0000","last_active":"2023-07-14T19:02:36.032950+0000","last_peered":"2023-07-14T19:02:36.032950+0000","last_clean":"2023-07-14T19:02:36.032950+0000","last_became_active":"2023-07-14T19:02:34.587202+0000","last_became_peered":"2023-07-14T19:02:34.587202+0000","last_unstale":"2023-07-14T19:02:36.032950+0000","last_undegraded":"2023-07-14T19:02:36.032950+0000","last_fullsized":"2023-07-14T19:02:36.032950+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:24.908235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.195784+0000","last_change":"2023-07-14T19:02:29.938027+0000","last_active":"2023-07-14T19:02:49.195784+0000","last_peered":"2023-07-14T19:02:49.195784+0000","last_clean":"2023-07-14T19:02:49.195784+0000","last_became_active":"2023-07-14T19:02:29.937690+0000","last_became_peered":"2023-07-14T19:02:29.937690+0000","last_unstale":"2023-07-14T19:02:49.195784+0000","last_undegraded":"2023-07-14T19:02:49.195784+0000","last_fullsized":"2023-07-14T19:02:49.195784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:03.527958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.207888+0000","last_change":"2023-07-14T19:02:30.576369+0000","last_active":"2023-07-14T19:03:30.207888+0000","last_peered":"2023-07-14T19:03:30.207888+0000","last_clean":"2023-07-14T19:03:30.207888+0000","last_became_active":"2023-07-14T19:02:30.576113+0000","last_became_peered":"2023-07-14T19:02:30.576113+0000","last_unstale":"2023-07-14T19:03:30.207888+0000","last_undegraded":"2023-07-14T19:03:30.207888+0000","last_fullsized":"2023-07-14T19:03:30.207888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:03:26.824693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213432+0000","last_change":"2023-07-14T19:02:34.583724+0000","last_active":"2023-07-14T19:03:31.213432+0000","last_peered":"2023-07-14T19:03:31.213432+0000","last_clean":"2023-07-14T19:03:31.213432+0000","last_became_active":"2023-07-14T19:02:34.583218+0000","last_became_peered":"2023-07-14T19:02:34.583218+0000","last_unstale":"2023-07-14T19:03:31.213432+0000","last_undegraded":"2023-07-14T19:03:31.213432+0000","last_fullsized":"2023-07-14T19:03:31.213432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:28:38.807293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608853+0000","last_change":"2023-07-14T19:02:29.938908+0000","last_active":"2023-07-14T19:02:35.608853+0000","last_peered":"2023-07-14T19:02:35.608853+0000","last_clean":"2023-07-14T19:02:35.608853+0000","last_became_active":"2023-07-14T19:02:29.938650+0000","last_became_peered":"2023-07-14T19:02:29.938650+0000","last_unstale":"2023-07-14T19:02:35.608853+0000","last_undegraded":"2023-07-14T19:02:35.608853+0000","last_fullsized":"2023-07-14T19:02:35.608853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:19.121855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811550+0000","last_change":"2023-07-14T19:02:30.571870+0000","last_active":"2023-07-14T19:02:35.811550+0000","last_peered":"2023-07-14T19:02:35.811550+0000","last_clean":"2023-07-14T19:02:35.811550+0000","last_became_active":"2023-07-14T19:02:30.571521+0000","last_became_peered":"2023-07-14T19:02:30.571521+0000","last_unstale":"2023-07-14T19:02:35.811550+0000","last_undegraded":"2023-07-14T19:02:35.811550+0000","last_fullsized":"2023-07-14T19:02:35.811550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:37.931915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213270+0000","last_change":"2023-07-14T19:02:34.589900+0000","last_active":"2023-07-14T19:03:31.213270+0000","last_peered":"2023-07-14T19:03:31.213270+0000","last_clean":"2023-07-14T19:03:31.213270+0000","last_became_active":"2023-07-14T19:02:34.589481+0000","last_became_peered":"2023-07-14T19:02:34.589481+0000","last_unstale":"2023-07-14T19:03:31.213270+0000","last_undegraded":"2023-07-14T19:03:31.213270+0000","last_fullsized":"2023-07-14T19:03:31.213270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:07.853098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206814+0000","last_change":"2023-07-14T19:02:29.940315+0000","last_active":"2023-07-14T19:03:30.206814+0000","last_peered":"2023-07-14T19:03:30.206814+0000","last_clean":"2023-07-14T19:03:30.206814+0000","last_became_active":"2023-07-14T19:02:29.939991+0000","last_became_peered":"2023-07-14T19:02:29.939991+0000","last_unstale":"2023-07-14T19:03:30.206814+0000","last_undegraded":"2023-07-14T19:03:30.206814+0000","last_fullsized":"2023-07-14T19:03:30.206814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:40.497517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214331+0000","last_change":"2023-07-14T19:02:30.583959+0000","last_active":"2023-07-14T19:03:30.214331+0000","last_peered":"2023-07-14T19:03:30.214331+0000","last_clean":"2023-07-14T19:03:30.214331+0000","last_became_active":"2023-07-14T19:02:30.583657+0000","last_became_peered":"2023-07-14T19:02:30.583657+0000","last_unstale":"2023-07-14T19:03:30.214331+0000","last_undegraded":"2023-07-14T19:03:30.214331+0000","last_fullsized":"2023-07-14T19:03:30.214331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:38:23.526665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.606896+0000","last_change":"2023-07-14T19:02:34.579328+0000","last_active":"2023-07-14T19:02:35.606896+0000","last_peered":"2023-07-14T19:02:35.606896+0000","last_clean":"2023-07-14T19:02:35.606896+0000","last_became_active":"2023-07-14T19:02:34.578904+0000","last_became_peered":"2023-07-14T19:02:34.578904+0000","last_unstale":"2023-07-14T19:02:35.606896+0000","last_undegraded":"2023-07-14T19:02:35.606896+0000","last_fullsized":"2023-07-14T19:02:35.606896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:26:48.233983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.207810+0000","last_change":"2023-07-14T19:02:29.940972+0000","last_active":"2023-07-14T19:02:50.207810+0000","last_peered":"2023-07-14T19:02:50.207810+0000","last_clean":"2023-07-14T19:02:50.207810+0000","last_became_active":"2023-07-14T19:02:29.940670+0000","last_became_peered":"2023-07-14T19:02:29.940670+0000","last_unstale":"2023-07-14T19:02:50.207810+0000","last_undegraded":"2023-07-14T19:02:50.207810+0000","last_fullsized":"2023-07-14T19:02:50.207810+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:09.182828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214674+0000","last_change":"2023-07-14T19:02:30.583714+0000","last_active":"2023-07-14T19:03:30.214674+0000","last_peered":"2023-07-14T19:03:30.214674+0000","last_clean":"2023-07-14T19:03:30.214674+0000","last_became_active":"2023-07-14T19:02:30.583404+0000","last_became_peered":"2023-07-14T19:02:30.583404+0000","last_unstale":"2023-07-14T19:03:30.214674+0000","last_undegraded":"2023-07-14T19:03:30.214674+0000","last_fullsized":"2023-07-14T19:03:30.214674+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:39:26.680582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207943+0000","last_change":"2023-07-14T19:02:34.582985+0000","last_active":"2023-07-14T19:03:31.207943+0000","last_peered":"2023-07-14T19:03:31.207943+0000","last_clean":"2023-07-14T19:03:31.207943+0000","last_became_active":"2023-07-14T19:02:34.582606+0000","last_became_peered":"2023-07-14T19:02:34.582606+0000","last_unstale":"2023-07-14T19:03:31.207943+0000","last_undegraded":"2023-07-14T19:03:31.207943+0000","last_fullsized":"2023-07-14T19:03:31.207943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:55:50.958196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219531+0000","last_change":"2023-07-14T19:02:29.944064+0000","last_active":"2023-07-14T19:03:31.219531+0000","last_peered":"2023-07-14T19:03:31.219531+0000","last_clean":"2023-07-14T19:03:31.219531+0000","last_became_active":"2023-07-14T19:02:29.943770+0000","last_became_peered":"2023-07-14T19:02:29.943770+0000","last_unstale":"2023-07-14T19:03:31.219531+0000","last_undegraded":"2023-07-14T19:03:31.219531+0000","last_fullsized":"2023-07-14T19:03:31.219531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:02:39.670163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214408+0000","last_change":"2023-07-14T19:02:30.580668+0000","last_active":"2023-07-14T19:03:30.214408+0000","last_peered":"2023-07-14T19:03:30.214408+0000","last_clean":"2023-07-14T19:03:30.214408+0000","last_became_active":"2023-07-14T19:02:30.580297+0000","last_became_peered":"2023-07-14T19:02:30.580297+0000","last_unstale":"2023-07-14T19:03:30.214408+0000","last_undegraded":"2023-07-14T19:03:30.214408+0000","last_fullsized":"2023-07-14T19:03:30.214408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:20.057488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.217879+0000","last_change":"2023-07-14T19:02:34.578894+0000","last_active":"2023-07-14T19:03:31.217879+0000","last_peered":"2023-07-14T19:03:31.217879+0000","last_clean":"2023-07-14T19:03:31.217879+0000","last_became_active":"2023-07-14T19:02:34.578463+0000","last_became_peered":"2023-07-14T19:02:34.578463+0000","last_unstale":"2023-07-14T19:03:31.217879+0000","last_undegraded":"2023-07-14T19:03:31.217879+0000","last_fullsized":"2023-07-14T19:03:31.217879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:53.046945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609171+0000","last_change":"2023-07-14T19:02:29.938430+0000","last_active":"2023-07-14T19:02:35.609171+0000","last_peered":"2023-07-14T19:02:35.609171+0000","last_clean":"2023-07-14T19:02:35.609171+0000","last_became_active":"2023-07-14T19:02:29.938118+0000","last_became_peered":"2023-07-14T19:02:29.938118+0000","last_unstale":"2023-07-14T19:02:35.609171+0000","last_undegraded":"2023-07-14T19:02:35.609171+0000","last_fullsized":"2023-07-14T19:02:35.609171+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:57:08.670837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214695+0000","last_change":"2023-07-14T19:02:30.575124+0000","last_active":"2023-07-14T19:03:30.214695+0000","last_peered":"2023-07-14T19:03:30.214695+0000","last_clean":"2023-07-14T19:03:30.214695+0000","last_became_active":"2023-07-14T19:02:30.574805+0000","last_became_peered":"2023-07-14T19:02:30.574805+0000","last_unstale":"2023-07-14T19:03:30.214695+0000","last_undegraded":"2023-07-14T19:03:30.214695+0000","last_fullsized":"2023-07-14T19:03:30.214695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:41.478433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208131+0000","last_change":"2023-07-14T19:02:30.573897+0000","last_active":"2023-07-14T19:03:30.208131+0000","last_peered":"2023-07-14T19:03:30.208131+0000","last_clean":"2023-07-14T19:03:30.208131+0000","last_became_active":"2023-07-14T19:02:30.573585+0000","last_became_peered":"2023-07-14T19:02:30.573585+0000","last_unstale":"2023-07-14T19:03:30.208131+0000","last_undegraded":"2023-07-14T19:03:30.208131+0000","last_fullsized":"2023-07-14T19:03:30.208131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:32:55.646800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608862+0000","last_change":"2023-07-14T19:02:30.572390+0000","last_active":"2023-07-14T19:02:35.608862+0000","last_peered":"2023-07-14T19:02:35.608862+0000","last_clean":"2023-07-14T19:02:35.608862+0000","last_became_active":"2023-07-14T19:02:30.572093+0000","last_became_peered":"2023-07-14T19:02:30.572093+0000","last_unstale":"2023-07-14T19:02:35.608862+0000","last_undegraded":"2023-07-14T19:02:35.608862+0000","last_fullsized":"2023-07-14T19:02:35.608862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:55:53.445808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608706+0000","last_change":"2023-07-14T19:02:30.572548+0000","last_active":"2023-07-14T19:02:35.608706+0000","last_peered":"2023-07-14T19:02:35.608706+0000","last_clean":"2023-07-14T19:02:35.608706+0000","last_became_active":"2023-07-14T19:02:30.572260+0000","last_became_peered":"2023-07-14T19:02:30.572260+0000","last_unstale":"2023-07-14T19:02:35.608706+0000","last_undegraded":"2023-07-14T19:02:35.608706+0000","last_fullsized":"2023-07-14T19:02:35.608706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:10.013952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208138+0000","last_change":"2023-07-14T19:02:30.569856+0000","last_active":"2023-07-14T19:03:31.208138+0000","last_peered":"2023-07-14T19:03:31.208138+0000","last_clean":"2023-07-14T19:03:31.208138+0000","last_became_active":"2023-07-14T19:02:30.569546+0000","last_became_peered":"2023-07-14T19:02:30.569546+0000","last_unstale":"2023-07-14T19:03:31.208138+0000","last_undegraded":"2023-07-14T19:03:31.208138+0000","last_fullsized":"2023-07-14T19:03:31.208138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:43.152189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608964+0000","last_change":"2023-07-14T19:02:30.570836+0000","last_active":"2023-07-14T19:02:35.608964+0000","last_peered":"2023-07-14T19:02:35.608964+0000","last_clean":"2023-07-14T19:02:35.608964+0000","last_became_active":"2023-07-14T19:02:30.569675+0000","last_became_peered":"2023-07-14T19:02:30.569675+0000","last_unstale":"2023-07-14T19:02:35.608964+0000","last_undegraded":"2023-07-14T19:02:35.608964+0000","last_fullsized":"2023-07-14T19:02:35.608964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:50.028159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.209264+0000","last_change":"2023-07-14T19:02:30.566846+0000","last_active":"2023-07-14T19:03:30.209264+0000","last_peered":"2023-07-14T19:03:30.209264+0000","last_clean":"2023-07-14T19:03:30.209264+0000","last_became_active":"2023-07-14T19:02:30.566525+0000","last_became_peered":"2023-07-14T19:02:30.566525+0000","last_unstale":"2023-07-14T19:03:30.209264+0000","last_undegraded":"2023-07-14T19:03:30.209264+0000","last_fullsized":"2023-07-14T19:03:30.209264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:59.170240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609207+0000","last_change":"2023-07-14T19:02:30.579680+0000","last_active":"2023-07-14T19:02:35.609207+0000","last_peered":"2023-07-14T19:02:35.609207+0000","last_clean":"2023-07-14T19:02:35.609207+0000","last_became_active":"2023-07-14T19:02:30.579379+0000","last_became_peered":"2023-07-14T19:02:30.579379+0000","last_unstale":"2023-07-14T19:02:35.609207+0000","last_undegraded":"2023-07-14T19:02:35.609207+0000","last_fullsized":"2023-07-14T19:02:35.609207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:37:18.570317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811962+0000","last_change":"2023-07-14T19:02:30.577722+0000","last_active":"2023-07-14T19:02:35.811962+0000","last_peered":"2023-07-14T19:02:35.811962+0000","last_clean":"2023-07-14T19:02:35.811962+0000","last_became_active":"2023-07-14T19:02:30.577247+0000","last_became_peered":"2023-07-14T19:02:30.577247+0000","last_unstale":"2023-07-14T19:02:35.811962+0000","last_undegraded":"2023-07-14T19:02:35.811962+0000","last_fullsized":"2023-07-14T19:02:35.811962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:29.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218436+0000","last_change":"2023-07-14T19:02:30.571052+0000","last_active":"2023-07-14T19:03:31.218436+0000","last_peered":"2023-07-14T19:03:31.218436+0000","last_clean":"2023-07-14T19:03:31.218436+0000","last_became_active":"2023-07-14T19:02:30.570722+0000","last_became_peered":"2023-07-14T19:02:30.570722+0000","last_unstale":"2023-07-14T19:03:31.218436+0000","last_undegraded":"2023-07-14T19:03:31.218436+0000","last_fullsized":"2023-07-14T19:03:31.218436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:50:37.718571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.812013+0000","last_change":"2023-07-14T19:02:30.575685+0000","last_active":"2023-07-14T19:02:35.812013+0000","last_peered":"2023-07-14T19:02:35.812013+0000","last_clean":"2023-07-14T19:02:35.812013+0000","last_became_active":"2023-07-14T19:02:30.575379+0000","last_became_peered":"2023-07-14T19:02:30.575379+0000","last_unstale":"2023-07-14T19:02:35.812013+0000","last_undegraded":"2023-07-14T19:02:35.812013+0000","last_fullsized":"2023-07-14T19:02:35.812013+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:31.763621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.607133+0000","last_change":"2023-07-14T19:02:30.582450+0000","last_active":"2023-07-14T19:02:35.607133+0000","last_peered":"2023-07-14T19:02:35.607133+0000","last_clean":"2023-07-14T19:02:35.607133+0000","last_became_active":"2023-07-14T19:02:30.582074+0000","last_became_peered":"2023-07-14T19:02:30.582074+0000","last_unstale":"2023-07-14T19:02:35.607133+0000","last_undegraded":"2023-07-14T19:02:35.607133+0000","last_fullsized":"2023-07-14T19:02:35.607133+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:17:46.572335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608673+0000","last_change":"2023-07-14T19:02:30.573042+0000","last_active":"2023-07-14T19:02:35.608673+0000","last_peered":"2023-07-14T19:02:35.608673+0000","last_clean":"2023-07-14T19:02:35.608673+0000","last_became_active":"2023-07-14T19:02:30.572498+0000","last_became_peered":"2023-07-14T19:02:30.572498+0000","last_unstale":"2023-07-14T19:02:35.608673+0000","last_undegraded":"2023-07-14T19:02:35.608673+0000","last_fullsized":"2023-07-14T19:02:35.608673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:42:33.413533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208888+0000","last_change":"2023-07-14T19:02:30.569441+0000","last_active":"2023-07-14T19:03:30.208888+0000","last_peered":"2023-07-14T19:03:30.208888+0000","last_clean":"2023-07-14T19:03:30.208888+0000","last_became_active":"2023-07-14T19:02:30.569180+0000","last_became_peered":"2023-07-14T19:02:30.569180+0000","last_unstale":"2023-07-14T19:03:30.208888+0000","last_undegraded":"2023-07-14T19:03:30.208888+0000","last_fullsized":"2023-07-14T19:03:30.208888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:34:29.430277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61264,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":12,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":264,"num_read_kb":194,"num_write":153,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514520,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":467112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.436},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.53900000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.622}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.76100000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.69099999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.81599999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.747}]}]},{"osd":10,"up_from":69,"seq":296352743451,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28600,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716552,"statfs":{"total":95995035648,"available":95965749248,"internally_reserved":0,"allocated":1622016,"data_stored":1102182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.67300000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.54100000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.63400000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.73999999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.98399999999999999,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.90900000000000003},"last":0.71099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"min":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"min":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.77700000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.81200000000000006},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.72599999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.83199999999999996},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.84799999999999998}]}]},{"osd":9,"up_from":64,"seq":274877906975,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":461934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.42099999999999999,"5min":0.63,"15min":0.63},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.23699999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.502},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79000000000000004,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.57999999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.71299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.64400000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47699999999999998,"15min":0.49299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66300000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.313},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.27200000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.41099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72499999999999998,"5min":1.008,"15min":1.008},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.746,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55800000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70499999999999996,"5min":1.026,"15min":1.026},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.59199999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.62},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.69999999999999996,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.81799999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.65500000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.73699999999999999,"15min":0.755},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81399999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84299999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.66600000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":0.97099999999999997,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":0.96099999999999997,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.69199999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":8,"up_from":57,"seq":244813135908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":457427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88300000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90900000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76300000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64700000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.878,"15min":0.878},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.76000000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.68400000000000005,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.88600000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.626,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.84899999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.94299999999999995,"5min":1.01,"15min":1.01},"last":0.65400000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.871,"15min":0.871},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0369999999999999,"5min":1.046,"15min":1.046},"last":0.82799999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.755,"15min":0.75},"min":{"1min":0.53200000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.626}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.98599999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]}]},{"osd":7,"up_from":50,"seq":214748364841,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1560576,"data_stored":1063197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56999999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.40899999999999997},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.65700000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57399999999999995,"15min":0.54900000000000004},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71699999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46200000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.47699999999999998,"15min":0.45900000000000002},"min":{"1min":0.38300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.48599999999999999},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69999999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.5}]},{"osd":5,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.627,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.87,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.752}]},{"osd":6,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.63},"min":{"1min":0.54100000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94899999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71599999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.79000000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74099999999999999,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96999999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.997,"5min":0.76300000000000001,"15min":0.72399999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":6.8970000000000002,"5min":6.8970000000000002,"15min":6.8970000000000002},"last":0.69399999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.72299999999999998},"min":{"1min":0.60599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95299999999999996,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.64700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.81399999999999995}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.83199999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":453014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39900000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.54700000000000004,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41999999999999998,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56200000000000006,"5min":1.591,"15min":1.591},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.55700000000000005},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83699999999999997,"5min":1.159,"15min":1.159},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":1.49,"15min":1.49},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73799999999999999,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61099999999999999,"15min":0.59299999999999997},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.77900000000000003,"5min":1.419,"15min":1.419},"last":0.41799999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.34300000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.48199999999999998,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":2.077,"15min":2.077},"last":0.35999999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85299999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64600000000000002,"15min":0.624},"min":{"1min":0.45500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80900000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65100000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76200000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.51200000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:02:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68600000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.76200000000000001,"15min":0.73499999999999999},"min":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.6709999999999998,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.69599999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73599999999999999},"min":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.77400000000000002,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78900000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.56299999999999994}]},{"osd":10,"last update":"Fri Jul 14 19:02:31 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.628},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.78900000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574913,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":509894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52400000000000002,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.3370000000000002},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.379,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":0.46100000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.55200000000000005,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.626,"5min":0.52500000000000002,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.47799999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88300000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81999999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.38600000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.44600000000000001,"15min":0.39100000000000001},"min":{"1min":0.36599999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44600000000000001,"15min":0.40200000000000002},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.35399999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.749},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55700000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61799999999999999,"15min":0.59199999999999997},"min":{"1min":0.53800000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.65000000000000002,"15min":0.60199999999999998},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.3209999999999997,"5min":4.3209999999999997,"15min":4.3209999999999997},"last":0.64000000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.52100000000000002,"15min":0.50800000000000001},"min":{"1min":0.32400000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.439},{"interface":"front","average":{"1min":0.67500000000000004,"5min":0.58299999999999996,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.78000000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72899999999999998,"15min":0.71599999999999997},"min":{"1min":0.498,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.99299999999999999,"5min":0.996,"15min":0.996},"last":0.69599999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.79000000000000004,"15min":0.77300000000000002},"min":{"1min":0.54800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.60099999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.59199999999999997}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.62}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83199999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":463639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.159,"15min":0.159},"max":{"1min":0.72499999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.48999999999999999},"min":{"1min":0.40300000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.75800000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.37,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.746,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.41299999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.435,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.29499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.32700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.372,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68700000000000006,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.627},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.50900000000000001},"min":{"1min":0.43099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.73899999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.46000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91700000000000004,"5min":1.042,"15min":1.042},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.496,"15min":0.495},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72099999999999997,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.67000000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60199999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70199999999999996,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.024,"15min":1.024},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.57999999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.78200000000000003,"5min":1.204,"15min":1.204},"last":0.503},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72499999999999998,"15min":0.746},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82199999999999995,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63900000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.877,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.79700000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.71499999999999997}]},{"osd":11,"last update":"Fri Jul 14 19:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66900000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":511586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78700000000000003,"5min":1.111,"15min":1.111},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.435}]},{"osd":1,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.71599999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77000000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55200000000000005,"15min":0.53900000000000003},"min":{"1min":0.45500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78100000000000003,"5min":1.046,"15min":1.046},"last":0.52900000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59699999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.44500000000000001,"15min":0.41799999999999998},"min":{"1min":0.33600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.59599999999999997,"15min":0.56599999999999995},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":1.4359999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60299999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89300000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.69599999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60199999999999998,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.748},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61599999999999999,"15min":0.60699999999999998},"min":{"1min":0.47499999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.79600000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.443,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1.204},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.59599999999999997,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.68600000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92600000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72799999999999998,"15min":0.71999999999999997},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.91300000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.81399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":453208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.188,"15min":0.188},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.253},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35299999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.215,"15min":0.215},"max":{"1min":0.52800000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.26900000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71799999999999997,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.497}]},{"osd":2,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.39600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.745,"5min":4.6509999999999998,"15min":4.6509999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60799999999999998,"15min":0.66400000000000003},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45700000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.498,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59099999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.82799999999999996,"5min":1.119,"15min":1.119},"last":0.33500000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76500000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.48999999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.60899999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66200000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76000000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.372}]},{"osd":10,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.626}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":476809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.503,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.38100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38400000000000001,"15min":0.34799999999999998},"min":{"1min":0.33600000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.59199999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.44},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40100000000000002,"15min":0.36499999999999999},"min":{"1min":0.34399999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.71799999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.376}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.89500000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.58499999999999996,"15min":0.54300000000000004},"min":{"1min":0.45500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.105,"15min":1.105},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55300000000000005,"15min":0.51200000000000001},"min":{"1min":0.432,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72299999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54900000000000004,"15min":0.50600000000000001},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48199999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57599999999999996,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.69599999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.60399999999999998},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":1.038,"15min":1.038},"last":0.59099999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.442,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.873,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68600000000000005,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.75600000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.58399999999999996,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77600000000000002}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.81000000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1067951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.495},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.41399999999999998,"15min":0.38900000000000001},"min":{"1min":0.309,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70599999999999996,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60199999999999998,"15min":0.58799999999999997},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84099999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64500000000000002,"15min":0.63100000000000001},"min":{"1min":0.40200000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.60699999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63900000000000001,"15min":0.622},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.65100000000000002,"15min":0.64000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.64700000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.71699999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.753},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.66300000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.755},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.73399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":58662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:03:33.549 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:03:33.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:33 smithi102 ceph-mon[129280]: pgmap v288: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:03:33.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:33 smithi088 ceph-mon[128973]: pgmap v288: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 329 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:03:34.119 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:34.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:34 smithi078 ceph-mon[127425]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:34.525 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:34 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:34 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:34 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.526 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:34 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:34 smithi102 ceph-mon[129280]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:34.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:34 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:34 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:34 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:34 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:34 smithi088 ceph-mon[128973]: from='client.15096 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:34.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:34 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:34 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:34 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:34.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:34 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:35.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 -- 172.21.15.78:0/3768859363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 msgr2=0x7f4404100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:35.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 --2- 172.21.15.78:0/3768859363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 0x7f4404100c10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f43f80099c0 tx=0x7f43f8009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 -- 172.21.15.78:0/3768859363 shutdown_connections 2023-07-14T19:03:35.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 --2- 172.21.15.78:0/3768859363 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4404101150 0x7f440410bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 --2- 172.21.15.78:0/3768859363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f44040fef40 0x7f44040ff310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 --2- 172.21.15.78:0/3768859363 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 0x7f4404100c10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 -- 172.21.15.78:0/3768859363 >> 172.21.15.78:0/3768859363 conn(0x7f4404075c90 msgr2=0x7f4404076090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:35.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 -- 172.21.15.78:0/3768859363 shutdown_connections 2023-07-14T19:03:35.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.340+0000 7f440a0d9700 1 -- 172.21.15.78:0/3768859363 wait complete. 2023-07-14T19:03:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.341+0000 7f440a0d9700 1 Processor -- start 2023-07-14T19:03:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f440a0d9700 1 -- start start 2023-07-14T19:03:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f440a0d9700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44040fef40 0x7f4404194d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f440a0d9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 0x7f44041952b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:35.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f440a0d9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404101150 0x7f4404199720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:35.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f440a0d9700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4404199d30 con 0x7f44041007a0 2023-07-14T19:03:35.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f440a0d9700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4404199ea0 con 0x7f44040fef40 2023-07-14T19:03:35.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f440a0d9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440419a060 con 0x7f4404101150 2023-07-14T19:03:35.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 0x7f44041952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:35.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 0x7f44041952b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35458/0 (socket says 172.21.15.78:35458) 2023-07-14T19:03:35.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 -- 172.21.15.78:0/2143501002 learned_addr learned my addr 172.21.15.78:0/2143501002 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:35.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f44037fe700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44040fef40 0x7f4404194d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:35.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4403fff700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404101150 0x7f4404199720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:35.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 -- 172.21.15.78:0/2143501002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404101150 msgr2=0x7f4404199720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:35.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404101150 0x7f4404199720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 -- 172.21.15.78:0/2143501002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44040fef40 msgr2=0x7f4404194d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:35.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44040fef40 0x7f4404194d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 -- 172.21.15.78:0/2143501002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f80095f0 con 0x7f44041007a0 2023-07-14T19:03:35.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.342+0000 7f4402ffd700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 0x7f44041952b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009f50 tx=0x7f43f8004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:35.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.343+0000 7f4400ff9700 1 -- 172.21.15.78:0/2143501002 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f801d070 con 0x7f44041007a0 2023-07-14T19:03:35.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.343+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440419a2e0 con 0x7f44041007a0 2023-07-14T19:03:35.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.343+0000 7f4400ff9700 1 -- 172.21.15.78:0/2143501002 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f43f8003fc0 con 0x7f44041007a0 2023-07-14T19:03:35.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.343+0000 7f4400ff9700 1 -- 172.21.15.78:0/2143501002 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f8022c30 con 0x7f44041007a0 2023-07-14T19:03:35.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.343+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4404068990 con 0x7f44041007a0 2023-07-14T19:03:35.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.344+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440404e340 con 0x7f44041007a0 2023-07-14T19:03:35.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.345+0000 7f4400ff9700 1 -- 172.21.15.78:0/2143501002 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f43f8022470 con 0x7f44041007a0 2023-07-14T19:03:35.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.345+0000 7f4400ff9700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f43ec0a11e0 0x7f43ec0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:35.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.346+0000 7f44037fe700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f43ec0a11e0 0x7f43ec0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:35.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.346+0000 7f4400ff9700 1 -- 172.21.15.78:0/2143501002 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f43f80ce660 con 0x7f44041007a0 2023-07-14T19:03:35.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.346+0000 7f44037fe700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f43ec0a11e0 0x7f43ec0a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f43f4005950 tx=0x7f43f400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:35.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.349+0000 7f4400ff9700 1 -- 172.21.15.78:0/2143501002 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f43f80979d0 con 0x7f44041007a0 2023-07-14T19:03:35.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.538+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f44040fcc60 con 0x7f43ec0a11e0 2023-07-14T19:03:35.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:35 smithi078 ceph-mon[127425]: pgmap v289: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T19:03:35.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:35.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:35 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:35.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.551+0000 7f4400ff9700 1 -- 172.21.15.78:0/2143501002 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+424434 (secure 0 0 0) 0x7f44040fcc60 con 0x7f43ec0a11e0 2023-07-14T19:03:35.559 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:35.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.554+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f43ec0a11e0 msgr2=0x7f43ec0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:35.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.554+0000 7f440a0d9700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f43ec0a11e0 0x7f43ec0a3690 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f43f4005950 tx=0x7f43f400b410 comp rx=0 tx=0).stop 2023-07-14T19:03:35.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 msgr2=0x7f44041952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 0x7f44041952b0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f43f8009f50 tx=0x7f43f8004710 comp rx=0 tx=0).stop 2023-07-14T19:03:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 shutdown_connections 2023-07-14T19:03:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f43ec0a11e0 0x7f43ec0a3690 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f44040fef40 0x7f4404194d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4404101150 0x7f4404199720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 --2- 172.21.15.78:0/2143501002 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f44041007a0 0x7f44041952b0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:35.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 >> 172.21.15.78:0/2143501002 conn(0x7f4404075c90 msgr2=0x7f44040fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:35.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 shutdown_connections 2023-07-14T19:03:35.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:35.555+0000 7f440a0d9700 1 -- 172.21.15.78:0/2143501002 wait complete. 2023-07-14T19:03:35.563 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-14T19:03:35.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:35 smithi102 ceph-mon[129280]: pgmap v289: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T19:03:35.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:35.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:35 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:35.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:35 smithi088 ceph-mon[128973]: pgmap v289: 129 pgs: 3 peering, 126 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s; 0 B/s, 0 objects/s recovering 2023-07-14T19:03:35.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:35.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:35 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:35.968 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":289,"stamp":"2023-07-14T19:03:34.125806+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651878,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":12,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":405,"num_read_kb":313,"num_write":336,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":422,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337496,"kb_used_data":13252,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604328,"statfs":{"total":1151940427776,"available":1151594831872,"internally_reserved":0,"allocated":13570048,"data_stored":7545181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":12,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":54,"num_read_kb":24,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002187"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.209253+0000","last_change":"2023-07-14T19:02:30.582427+0000","last_active":"2023-07-14T19:03:30.209253+0000","last_peered":"2023-07-14T19:03:30.209253+0000","last_clean":"2023-07-14T19:03:30.209253+0000","last_became_active":"2023-07-14T19:02:30.582149+0000","last_became_peered":"2023-07-14T19:02:30.582149+0000","last_unstale":"2023-07-14T19:03:30.209253+0000","last_undegraded":"2023-07-14T19:03:30.209253+0000","last_fullsized":"2023-07-14T19:03:30.209253+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:43:06.312716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208324+0000","last_change":"2023-07-14T19:02:30.567053+0000","last_active":"2023-07-14T19:03:31.208324+0000","last_peered":"2023-07-14T19:03:31.208324+0000","last_clean":"2023-07-14T19:03:31.208324+0000","last_became_active":"2023-07-14T19:02:30.566736+0000","last_became_peered":"2023-07-14T19:02:30.566736+0000","last_unstale":"2023-07-14T19:03:31.208324+0000","last_undegraded":"2023-07-14T19:03:31.208324+0000","last_fullsized":"2023-07-14T19:03:31.208324+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:44:23.099904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208244+0000","last_change":"2023-07-14T19:02:30.577632+0000","last_active":"2023-07-14T19:03:31.208244+0000","last_peered":"2023-07-14T19:03:31.208244+0000","last_clean":"2023-07-14T19:03:31.208244+0000","last_became_active":"2023-07-14T19:02:30.577300+0000","last_became_peered":"2023-07-14T19:02:30.577300+0000","last_unstale":"2023-07-14T19:03:31.208244+0000","last_undegraded":"2023-07-14T19:03:31.208244+0000","last_fullsized":"2023-07-14T19:03:31.208244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:15:18.165242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213866+0000","last_change":"2023-07-14T19:02:30.581405+0000","last_active":"2023-07-14T19:03:31.213866+0000","last_peered":"2023-07-14T19:03:31.213866+0000","last_clean":"2023-07-14T19:03:31.213866+0000","last_became_active":"2023-07-14T19:02:30.581113+0000","last_became_peered":"2023-07-14T19:02:30.581113+0000","last_unstale":"2023-07-14T19:03:31.213866+0000","last_undegraded":"2023-07-14T19:03:31.213866+0000","last_fullsized":"2023-07-14T19:03:31.213866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:42:53.074489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206313+0000","last_change":"2023-07-14T19:02:30.574285+0000","last_active":"2023-07-14T19:03:30.206313+0000","last_peered":"2023-07-14T19:03:30.206313+0000","last_clean":"2023-07-14T19:03:30.206313+0000","last_became_active":"2023-07-14T19:02:30.573967+0000","last_became_peered":"2023-07-14T19:02:30.573967+0000","last_unstale":"2023-07-14T19:03:30.206313+0000","last_undegraded":"2023-07-14T19:03:30.206313+0000","last_fullsized":"2023-07-14T19:03:30.206313+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:42:45.695145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215131+0000","last_change":"2023-07-14T19:02:30.574135+0000","last_active":"2023-07-14T19:03:31.215131+0000","last_peered":"2023-07-14T19:03:31.215131+0000","last_clean":"2023-07-14T19:03:31.215131+0000","last_became_active":"2023-07-14T19:02:30.573821+0000","last_became_peered":"2023-07-14T19:02:30.573821+0000","last_unstale":"2023-07-14T19:03:31.215131+0000","last_undegraded":"2023-07-14T19:03:31.215131+0000","last_fullsized":"2023-07-14T19:03:31.215131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:56.064229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.213960+0000","last_change":"2023-07-14T19:02:30.578051+0000","last_active":"2023-07-14T19:03:30.213960+0000","last_peered":"2023-07-14T19:03:30.213960+0000","last_clean":"2023-07-14T19:03:30.213960+0000","last_became_active":"2023-07-14T19:02:30.577700+0000","last_became_peered":"2023-07-14T19:02:30.577700+0000","last_unstale":"2023-07-14T19:03:30.213960+0000","last_undegraded":"2023-07-14T19:03:30.213960+0000","last_fullsized":"2023-07-14T19:03:30.213960+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:31:57.113901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811504+0000","last_change":"2023-07-14T19:02:30.567480+0000","last_active":"2023-07-14T19:02:35.811504+0000","last_peered":"2023-07-14T19:02:35.811504+0000","last_clean":"2023-07-14T19:02:35.811504+0000","last_became_active":"2023-07-14T19:02:30.567147+0000","last_became_peered":"2023-07-14T19:02:30.567147+0000","last_unstale":"2023-07-14T19:02:35.811504+0000","last_undegraded":"2023-07-14T19:02:35.811504+0000","last_fullsized":"2023-07-14T19:02:35.811504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:27:53.531353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281462+0000","last_change":"2023-07-14T19:02:30.581936+0000","last_active":"2023-07-14T19:03:31.281462+0000","last_peered":"2023-07-14T19:03:31.281462+0000","last_clean":"2023-07-14T19:03:31.281462+0000","last_became_active":"2023-07-14T19:02:30.579619+0000","last_became_peered":"2023-07-14T19:02:30.579619+0000","last_unstale":"2023-07-14T19:03:31.281462+0000","last_undegraded":"2023-07-14T19:03:31.281462+0000","last_fullsized":"2023-07-14T19:03:31.281462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:23.438941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218216+0000","last_change":"2023-07-14T19:03:31.218216+0000","last_active":"2023-07-14T19:03:31.218216+0000","last_peered":"2023-07-14T19:03:31.218216+0000","last_clean":"2023-07-14T19:03:31.218216+0000","last_became_active":"2023-07-14T19:03:31.217871+0000","last_became_peered":"2023-07-14T19:03:31.217871+0000","last_unstale":"2023-07-14T19:03:31.218216+0000","last_undegraded":"2023-07-14T19:03:31.218216+0000","last_fullsized":"2023-07-14T19:03:31.218216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:56:32.678369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214597+0000","last_change":"2023-07-14T19:02:30.583662+0000","last_active":"2023-07-14T19:03:30.214597+0000","last_peered":"2023-07-14T19:03:30.214597+0000","last_clean":"2023-07-14T19:03:30.214597+0000","last_became_active":"2023-07-14T19:02:30.583352+0000","last_became_peered":"2023-07-14T19:02:30.583352+0000","last_unstale":"2023-07-14T19:03:30.214597+0000","last_undegraded":"2023-07-14T19:03:30.214597+0000","last_fullsized":"2023-07-14T19:03:30.214597+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:11:29.149234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213712+0000","last_change":"2023-07-14T19:02:30.567685+0000","last_active":"2023-07-14T19:03:31.213712+0000","last_peered":"2023-07-14T19:03:31.213712+0000","last_clean":"2023-07-14T19:03:31.213712+0000","last_became_active":"2023-07-14T19:02:30.566674+0000","last_became_peered":"2023-07-14T19:02:30.566674+0000","last_unstale":"2023-07-14T19:03:31.213712+0000","last_undegraded":"2023-07-14T19:03:31.213712+0000","last_fullsized":"2023-07-14T19:03:31.213712+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:41:27.460945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608702+0000","last_change":"2023-07-14T19:02:30.558012+0000","last_active":"2023-07-14T19:02:35.608702+0000","last_peered":"2023-07-14T19:02:35.608702+0000","last_clean":"2023-07-14T19:02:35.608702+0000","last_became_active":"2023-07-14T19:02:30.557675+0000","last_became_peered":"2023-07-14T19:02:30.557675+0000","last_unstale":"2023-07-14T19:02:35.608702+0000","last_undegraded":"2023-07-14T19:02:35.608702+0000","last_fullsized":"2023-07-14T19:02:35.608702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:56.276352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207948+0000","last_change":"2023-07-14T19:02:30.570341+0000","last_active":"2023-07-14T19:03:31.207948+0000","last_peered":"2023-07-14T19:03:31.207948+0000","last_clean":"2023-07-14T19:03:31.207948+0000","last_became_active":"2023-07-14T19:02:30.570091+0000","last_became_peered":"2023-07-14T19:02:30.570091+0000","last_unstale":"2023-07-14T19:03:31.207948+0000","last_undegraded":"2023-07-14T19:03:31.207948+0000","last_fullsized":"2023-07-14T19:03:31.207948+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:52:13.273335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207886+0000","last_change":"2023-07-14T19:02:30.573746+0000","last_active":"2023-07-14T19:03:31.207886+0000","last_peered":"2023-07-14T19:03:31.207886+0000","last_clean":"2023-07-14T19:03:31.207886+0000","last_became_active":"2023-07-14T19:02:30.573307+0000","last_became_peered":"2023-07-14T19:02:30.573307+0000","last_unstale":"2023-07-14T19:03:31.207886+0000","last_undegraded":"2023-07-14T19:03:31.207886+0000","last_fullsized":"2023-07-14T19:03:31.207886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:49.299962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.280974+0000","last_change":"2023-07-14T19:02:30.578352+0000","last_active":"2023-07-14T19:03:31.280974+0000","last_peered":"2023-07-14T19:03:31.280974+0000","last_clean":"2023-07-14T19:03:31.280974+0000","last_became_active":"2023-07-14T19:02:30.577631+0000","last_became_peered":"2023-07-14T19:02:30.577631+0000","last_unstale":"2023-07-14T19:03:31.280974+0000","last_undegraded":"2023-07-14T19:03:31.280974+0000","last_fullsized":"2023-07-14T19:03:31.280974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:07:01.573189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.209130+0000","last_change":"2023-07-14T19:02:30.577813+0000","last_active":"2023-07-14T19:03:30.209130+0000","last_peered":"2023-07-14T19:03:30.209130+0000","last_clean":"2023-07-14T19:03:30.209130+0000","last_became_active":"2023-07-14T19:02:30.577482+0000","last_became_peered":"2023-07-14T19:02:30.577482+0000","last_unstale":"2023-07-14T19:03:30.209130+0000","last_undegraded":"2023-07-14T19:03:30.209130+0000","last_fullsized":"2023-07-14T19:03:30.209130+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:22:56.256630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210222+0000","last_change":"2023-07-14T19:02:30.583353+0000","last_active":"2023-07-14T19:03:31.210222+0000","last_peered":"2023-07-14T19:03:31.210222+0000","last_clean":"2023-07-14T19:03:31.210222+0000","last_became_active":"2023-07-14T19:02:30.583037+0000","last_became_peered":"2023-07-14T19:02:30.583037+0000","last_unstale":"2023-07-14T19:03:31.210222+0000","last_undegraded":"2023-07-14T19:03:31.210222+0000","last_fullsized":"2023-07-14T19:03:31.210222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:43:40.021757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609345+0000","last_change":"2023-07-14T19:02:30.569788+0000","last_active":"2023-07-14T19:02:35.609345+0000","last_peered":"2023-07-14T19:02:35.609345+0000","last_clean":"2023-07-14T19:02:35.609345+0000","last_became_active":"2023-07-14T19:02:30.569461+0000","last_became_peered":"2023-07-14T19:02:30.569461+0000","last_unstale":"2023-07-14T19:02:35.609345+0000","last_undegraded":"2023-07-14T19:02:35.609345+0000","last_fullsized":"2023-07-14T19:02:35.609345+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:35:17.252918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608767+0000","last_change":"2023-07-14T19:02:34.585220+0000","last_active":"2023-07-14T19:02:35.608767+0000","last_peered":"2023-07-14T19:02:35.608767+0000","last_clean":"2023-07-14T19:02:35.608767+0000","last_became_active":"2023-07-14T19:02:34.584771+0000","last_became_peered":"2023-07-14T19:02:34.584771+0000","last_unstale":"2023-07-14T19:02:35.608767+0000","last_undegraded":"2023-07-14T19:02:35.608767+0000","last_fullsized":"2023-07-14T19:02:35.608767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:53:48.195878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210698+0000","last_change":"2023-07-14T19:02:29.938571+0000","last_active":"2023-07-14T19:03:31.210698+0000","last_peered":"2023-07-14T19:03:31.210698+0000","last_clean":"2023-07-14T19:03:31.210698+0000","last_became_active":"2023-07-14T19:02:29.938270+0000","last_became_peered":"2023-07-14T19:02:29.938270+0000","last_unstale":"2023-07-14T19:03:31.210698+0000","last_undegraded":"2023-07-14T19:03:31.210698+0000","last_fullsized":"2023-07-14T19:03:31.210698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:02.148507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811982+0000","last_change":"2023-07-14T19:02:30.575894+0000","last_active":"2023-07-14T19:02:35.811982+0000","last_peered":"2023-07-14T19:02:35.811982+0000","last_clean":"2023-07-14T19:02:35.811982+0000","last_became_active":"2023-07-14T19:02:30.575607+0000","last_became_peered":"2023-07-14T19:02:30.575607+0000","last_unstale":"2023-07-14T19:02:35.811982+0000","last_undegraded":"2023-07-14T19:02:35.811982+0000","last_fullsized":"2023-07-14T19:02:35.811982+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:06:30.612104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218192+0000","last_change":"2023-07-14T19:03:31.218192+0000","last_active":"2023-07-14T19:03:31.218192+0000","last_peered":"2023-07-14T19:03:31.218192+0000","last_clean":"2023-07-14T19:03:31.218192+0000","last_became_active":"2023-07-14T19:03:31.217708+0000","last_became_peered":"2023-07-14T19:03:31.217708+0000","last_unstale":"2023-07-14T19:03:31.218192+0000","last_undegraded":"2023-07-14T19:03:31.218192+0000","last_fullsized":"2023-07-14T19:03:31.218192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:40.671215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.812123+0000","last_change":"2023-07-14T19:02:29.944136+0000","last_active":"2023-07-14T19:02:35.812123+0000","last_peered":"2023-07-14T19:02:35.812123+0000","last_clean":"2023-07-14T19:02:35.812123+0000","last_became_active":"2023-07-14T19:02:29.943751+0000","last_became_peered":"2023-07-14T19:02:29.943751+0000","last_unstale":"2023-07-14T19:02:35.812123+0000","last_undegraded":"2023-07-14T19:02:35.812123+0000","last_fullsized":"2023-07-14T19:02:35.812123+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:34:19.730141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208280+0000","last_change":"2023-07-14T19:02:30.583156+0000","last_active":"2023-07-14T19:03:31.208280+0000","last_peered":"2023-07-14T19:03:31.208280+0000","last_clean":"2023-07-14T19:03:31.208280+0000","last_became_active":"2023-07-14T19:02:30.582794+0000","last_became_peered":"2023-07-14T19:02:30.582794+0000","last_unstale":"2023-07-14T19:03:31.208280+0000","last_undegraded":"2023-07-14T19:03:31.208280+0000","last_fullsized":"2023-07-14T19:03:31.208280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:47:47.558145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214058+0000","last_change":"2023-07-14T19:02:34.590788+0000","last_active":"2023-07-14T19:03:31.214058+0000","last_peered":"2023-07-14T19:03:31.214058+0000","last_clean":"2023-07-14T19:03:31.214058+0000","last_became_active":"2023-07-14T19:02:34.590385+0000","last_became_peered":"2023-07-14T19:02:34.590385+0000","last_unstale":"2023-07-14T19:03:31.214058+0000","last_undegraded":"2023-07-14T19:03:31.214058+0000","last_fullsized":"2023-07-14T19:03:31.214058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:35.914846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":27,"reported_epoch":85,"state":"peering","last_fresh":"2023-07-14T19:02:50.208789+0000","last_change":"2023-07-14T19:03:30.207818+0000","last_active":"2023-07-14T19:03:30.207818+0000","last_peered":"2023-07-14T19:02:50.208789+0000","last_clean":"2023-07-14T19:02:50.208789+0000","last_became_active":"2023-07-14T19:02:29.937823+0000","last_became_peered":"2023-07-14T19:02:29.937823+0000","last_unstale":"2023-07-14T19:02:50.208789+0000","last_undegraded":"2023-07-14T19:02:50.208789+0000","last_fullsized":"2023-07-14T19:02:50.208789+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281375+0000","last_change":"2023-07-14T19:02:30.582746+0000","last_active":"2023-07-14T19:03:31.281375+0000","last_peered":"2023-07-14T19:03:31.281375+0000","last_clean":"2023-07-14T19:03:31.281375+0000","last_became_active":"2023-07-14T19:02:30.582426+0000","last_became_peered":"2023-07-14T19:02:30.582426+0000","last_unstale":"2023-07-14T19:03:31.281375+0000","last_undegraded":"2023-07-14T19:03:31.281375+0000","last_fullsized":"2023-07-14T19:03:31.281375+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:16:53.120645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212948+0000","last_change":"2023-07-14T19:02:34.589966+0000","last_active":"2023-07-14T19:03:31.212948+0000","last_peered":"2023-07-14T19:03:31.212948+0000","last_clean":"2023-07-14T19:03:31.212948+0000","last_became_active":"2023-07-14T19:02:34.589544+0000","last_became_peered":"2023-07-14T19:02:34.589544+0000","last_unstale":"2023-07-14T19:03:31.212948+0000","last_undegraded":"2023-07-14T19:03:31.212948+0000","last_fullsized":"2023-07-14T19:03:31.212948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:08.813030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206624+0000","last_change":"2023-07-14T19:02:29.935507+0000","last_active":"2023-07-14T19:03:30.206624+0000","last_peered":"2023-07-14T19:03:30.206624+0000","last_clean":"2023-07-14T19:03:30.206624+0000","last_became_active":"2023-07-14T19:02:29.935160+0000","last_became_peered":"2023-07-14T19:02:29.935160+0000","last_unstale":"2023-07-14T19:03:30.206624+0000","last_undegraded":"2023-07-14T19:03:30.206624+0000","last_fullsized":"2023-07-14T19:03:30.206624+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:36:48.126803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219342+0000","last_change":"2023-07-14T19:02:30.578077+0000","last_active":"2023-07-14T19:03:31.219342+0000","last_peered":"2023-07-14T19:03:31.219342+0000","last_clean":"2023-07-14T19:03:31.219342+0000","last_became_active":"2023-07-14T19:02:30.577764+0000","last_became_peered":"2023-07-14T19:02:30.577764+0000","last_unstale":"2023-07-14T19:03:31.219342+0000","last_undegraded":"2023-07-14T19:03:31.219342+0000","last_fullsized":"2023-07-14T19:03:31.219342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:40:14.188240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609074+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:02:35.609074+0000","last_peered":"2023-07-14T19:02:35.609074+0000","last_clean":"2023-07-14T19:02:35.609074+0000","last_became_active":"2023-07-14T19:02:34.583408+0000","last_became_peered":"2023-07-14T19:02:34.583408+0000","last_unstale":"2023-07-14T19:02:35.609074+0000","last_undegraded":"2023-07-14T19:02:35.609074+0000","last_fullsized":"2023-07-14T19:02:35.609074+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:56:32.640321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.220191+0000","last_change":"2023-07-14T19:02:29.943899+0000","last_active":"2023-07-14T19:03:31.220191+0000","last_peered":"2023-07-14T19:03:31.220191+0000","last_clean":"2023-07-14T19:03:31.220191+0000","last_became_active":"2023-07-14T19:02:29.943585+0000","last_became_peered":"2023-07-14T19:02:29.943585+0000","last_unstale":"2023-07-14T19:03:31.220191+0000","last_undegraded":"2023-07-14T19:03:31.220191+0000","last_fullsized":"2023-07-14T19:03:31.220191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:29.940165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.207480+0000","last_change":"2023-07-14T19:02:30.567147+0000","last_active":"2023-07-14T19:03:30.207480+0000","last_peered":"2023-07-14T19:03:30.207480+0000","last_clean":"2023-07-14T19:03:30.207480+0000","last_became_active":"2023-07-14T19:02:30.566792+0000","last_became_peered":"2023-07-14T19:02:30.566792+0000","last_unstale":"2023-07-14T19:03:30.207480+0000","last_undegraded":"2023-07-14T19:03:30.207480+0000","last_fullsized":"2023-07-14T19:03:30.207480+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:56.705016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609090+0000","last_change":"2023-07-14T19:02:34.579561+0000","last_active":"2023-07-14T19:02:35.609090+0000","last_peered":"2023-07-14T19:02:35.609090+0000","last_clean":"2023-07-14T19:02:35.609090+0000","last_became_active":"2023-07-14T19:02:34.579099+0000","last_became_peered":"2023-07-14T19:02:34.579099+0000","last_unstale":"2023-07-14T19:02:35.609090+0000","last_undegraded":"2023-07-14T19:02:35.609090+0000","last_fullsized":"2023-07-14T19:02:35.609090+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:24:00.439199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.284881+0000","last_change":"2023-07-14T19:03:31.284881+0000","last_active":"2023-07-14T19:03:31.284881+0000","last_peered":"2023-07-14T19:03:31.284881+0000","last_clean":"2023-07-14T19:03:31.284881+0000","last_became_active":"2023-07-14T19:03:31.282061+0000","last_became_peered":"2023-07-14T19:03:31.282061+0000","last_unstale":"2023-07-14T19:03:31.284881+0000","last_undegraded":"2023-07-14T19:03:31.284881+0000","last_fullsized":"2023-07-14T19:03:31.284881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:32.928291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.220085+0000","last_change":"2023-07-14T19:02:30.575925+0000","last_active":"2023-07-14T19:03:31.220085+0000","last_peered":"2023-07-14T19:03:31.220085+0000","last_clean":"2023-07-14T19:03:31.220085+0000","last_became_active":"2023-07-14T19:02:30.575583+0000","last_became_peered":"2023-07-14T19:02:30.575583+0000","last_unstale":"2023-07-14T19:03:31.220085+0000","last_undegraded":"2023-07-14T19:03:31.220085+0000","last_fullsized":"2023-07-14T19:03:31.220085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:11:02.245125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208976+0000","last_change":"2023-07-14T19:02:34.587094+0000","last_active":"2023-07-14T19:03:30.208976+0000","last_peered":"2023-07-14T19:03:30.208976+0000","last_clean":"2023-07-14T19:03:30.208976+0000","last_became_active":"2023-07-14T19:02:34.586650+0000","last_became_peered":"2023-07-14T19:02:34.586650+0000","last_unstale":"2023-07-14T19:03:30.208976+0000","last_undegraded":"2023-07-14T19:03:30.208976+0000","last_fullsized":"2023-07-14T19:03:30.208976+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:59:14.503188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.209484+0000","last_change":"2023-07-14T19:02:29.934651+0000","last_active":"2023-07-14T19:02:50.209484+0000","last_peered":"2023-07-14T19:02:50.209484+0000","last_clean":"2023-07-14T19:02:50.209484+0000","last_became_active":"2023-07-14T19:02:29.934347+0000","last_became_peered":"2023-07-14T19:02:29.934347+0000","last_unstale":"2023-07-14T19:02:50.209484+0000","last_undegraded":"2023-07-14T19:02:50.209484+0000","last_fullsized":"2023-07-14T19:02:50.209484+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:34:23.476531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811237+0000","last_change":"2023-07-14T19:02:30.571680+0000","last_active":"2023-07-14T19:02:35.811237+0000","last_peered":"2023-07-14T19:02:35.811237+0000","last_clean":"2023-07-14T19:02:35.811237+0000","last_became_active":"2023-07-14T19:02:30.571367+0000","last_became_peered":"2023-07-14T19:02:30.571367+0000","last_unstale":"2023-07-14T19:02:35.811237+0000","last_undegraded":"2023-07-14T19:02:35.811237+0000","last_fullsized":"2023-07-14T19:02:35.811237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:43.958053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208272+0000","last_change":"2023-07-14T19:02:34.577642+0000","last_active":"2023-07-14T19:03:31.208272+0000","last_peered":"2023-07-14T19:03:31.208272+0000","last_clean":"2023-07-14T19:03:31.208272+0000","last_became_active":"2023-07-14T19:02:34.577232+0000","last_became_peered":"2023-07-14T19:02:34.577232+0000","last_unstale":"2023-07-14T19:03:31.208272+0000","last_undegraded":"2023-07-14T19:03:31.208272+0000","last_fullsized":"2023-07-14T19:03:31.208272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:19.952531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218302+0000","last_change":"2023-07-14T19:02:29.940688+0000","last_active":"2023-07-14T19:03:31.218302+0000","last_peered":"2023-07-14T19:03:31.218302+0000","last_clean":"2023-07-14T19:03:31.218302+0000","last_became_active":"2023-07-14T19:02:29.940444+0000","last_became_peered":"2023-07-14T19:02:29.940444+0000","last_unstale":"2023-07-14T19:03:31.218302+0000","last_undegraded":"2023-07-14T19:03:31.218302+0000","last_fullsized":"2023-07-14T19:03:31.218302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:37.769743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210635+0000","last_change":"2023-07-14T19:02:30.568127+0000","last_active":"2023-07-14T19:03:31.210635+0000","last_peered":"2023-07-14T19:03:31.210635+0000","last_clean":"2023-07-14T19:03:31.210635+0000","last_became_active":"2023-07-14T19:02:30.567811+0000","last_became_peered":"2023-07-14T19:02:30.567811+0000","last_unstale":"2023-07-14T19:03:31.210635+0000","last_undegraded":"2023-07-14T19:03:31.210635+0000","last_fullsized":"2023-07-14T19:03:31.210635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:56.545251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811843+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:02:35.811843+0000","last_peered":"2023-07-14T19:02:35.811843+0000","last_clean":"2023-07-14T19:02:35.811843+0000","last_became_active":"2023-07-14T19:02:34.583480+0000","last_became_peered":"2023-07-14T19:02:34.583480+0000","last_unstale":"2023-07-14T19:02:35.811843+0000","last_undegraded":"2023-07-14T19:02:35.811843+0000","last_fullsized":"2023-07-14T19:02:35.811843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:27.737212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.677712+0000","last_change":"2023-07-14T19:02:29.938459+0000","last_active":"2023-07-14T19:03:31.677712+0000","last_peered":"2023-07-14T19:03:31.677712+0000","last_clean":"2023-07-14T19:03:31.677712+0000","last_became_active":"2023-07-14T19:02:29.938212+0000","last_became_peered":"2023-07-14T19:02:29.938212+0000","last_unstale":"2023-07-14T19:03:31.677712+0000","last_undegraded":"2023-07-14T19:03:31.677712+0000","last_fullsized":"2023-07-14T19:03:31.677712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:34.199818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281471+0000","last_change":"2023-07-14T19:02:30.586345+0000","last_active":"2023-07-14T19:03:31.281471+0000","last_peered":"2023-07-14T19:03:31.281471+0000","last_clean":"2023-07-14T19:03:31.281471+0000","last_became_active":"2023-07-14T19:02:30.585949+0000","last_became_peered":"2023-07-14T19:02:30.585949+0000","last_unstale":"2023-07-14T19:03:31.281471+0000","last_undegraded":"2023-07-14T19:03:31.281471+0000","last_fullsized":"2023-07-14T19:03:31.281471+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:50:32.754337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210309+0000","last_change":"2023-07-14T19:02:34.582922+0000","last_active":"2023-07-14T19:03:31.210309+0000","last_peered":"2023-07-14T19:03:31.210309+0000","last_clean":"2023-07-14T19:03:31.210309+0000","last_became_active":"2023-07-14T19:02:34.582146+0000","last_became_peered":"2023-07-14T19:02:34.582146+0000","last_unstale":"2023-07-14T19:03:31.210309+0000","last_undegraded":"2023-07-14T19:03:31.210309+0000","last_fullsized":"2023-07-14T19:03:31.210309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:15.121038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214601+0000","last_change":"2023-07-14T19:02:29.938215+0000","last_active":"2023-07-14T19:03:30.214601+0000","last_peered":"2023-07-14T19:03:30.214601+0000","last_clean":"2023-07-14T19:03:30.214601+0000","last_became_active":"2023-07-14T19:02:29.937889+0000","last_became_peered":"2023-07-14T19:02:29.937889+0000","last_unstale":"2023-07-14T19:03:30.214601+0000","last_undegraded":"2023-07-14T19:03:30.214601+0000","last_fullsized":"2023-07-14T19:03:30.214601+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:18:51.892537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208284+0000","last_change":"2023-07-14T19:02:30.569622+0000","last_active":"2023-07-14T19:03:31.208284+0000","last_peered":"2023-07-14T19:03:31.208284+0000","last_clean":"2023-07-14T19:03:31.208284+0000","last_became_active":"2023-07-14T19:02:30.569201+0000","last_became_peered":"2023-07-14T19:02:30.569201+0000","last_unstale":"2023-07-14T19:03:31.208284+0000","last_undegraded":"2023-07-14T19:03:31.208284+0000","last_fullsized":"2023-07-14T19:03:31.208284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:53:07.451914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214242+0000","last_change":"2023-07-14T19:02:34.580978+0000","last_active":"2023-07-14T19:03:31.214242+0000","last_peered":"2023-07-14T19:03:31.214242+0000","last_clean":"2023-07-14T19:03:31.214242+0000","last_became_active":"2023-07-14T19:02:34.580503+0000","last_became_peered":"2023-07-14T19:02:34.580503+0000","last_unstale":"2023-07-14T19:03:31.214242+0000","last_undegraded":"2023-07-14T19:03:31.214242+0000","last_fullsized":"2023-07-14T19:03:31.214242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:06.819824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219409+0000","last_change":"2023-07-14T19:02:29.942400+0000","last_active":"2023-07-14T19:03:31.219409+0000","last_peered":"2023-07-14T19:03:31.219409+0000","last_clean":"2023-07-14T19:03:31.219409+0000","last_became_active":"2023-07-14T19:02:29.942080+0000","last_became_peered":"2023-07-14T19:02:29.942080+0000","last_unstale":"2023-07-14T19:03:31.219409+0000","last_undegraded":"2023-07-14T19:03:31.219409+0000","last_fullsized":"2023-07-14T19:03:31.219409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:09:06.130256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281477+0000","last_change":"2023-07-14T19:02:30.582869+0000","last_active":"2023-07-14T19:03:31.281477+0000","last_peered":"2023-07-14T19:03:31.281477+0000","last_clean":"2023-07-14T19:03:31.281477+0000","last_became_active":"2023-07-14T19:02:30.582558+0000","last_became_peered":"2023-07-14T19:02:30.582558+0000","last_unstale":"2023-07-14T19:03:31.281477+0000","last_undegraded":"2023-07-14T19:03:31.281477+0000","last_fullsized":"2023-07-14T19:03:31.281477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:51:36.705425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.216738+0000","last_change":"2023-07-14T19:03:31.216738+0000","last_active":"2023-07-14T19:03:31.216738+0000","last_peered":"2023-07-14T19:03:31.216738+0000","last_clean":"2023-07-14T19:03:31.216738+0000","last_became_active":"2023-07-14T19:03:31.216310+0000","last_became_peered":"2023-07-14T19:03:31.216310+0000","last_unstale":"2023-07-14T19:03:31.216738+0000","last_undegraded":"2023-07-14T19:03:31.216738+0000","last_fullsized":"2023-07-14T19:03:31.216738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:26.359586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'30","reported_seq":149,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:32.686954+0000","last_change":"2023-07-14T19:02:29.939314+0000","last_active":"2023-07-14T19:03:32.686954+0000","last_peered":"2023-07-14T19:03:32.686954+0000","last_clean":"2023-07-14T19:03:32.686954+0000","last_became_active":"2023-07-14T19:02:29.939000+0000","last_became_peered":"2023-07-14T19:02:29.939000+0000","last_unstale":"2023-07-14T19:03:32.686954+0000","last_undegraded":"2023-07-14T19:03:32.686954+0000","last_fullsized":"2023-07-14T19:03:32.686954+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:43.443151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56970,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":93,"num_read_kb":102,"num_write":31,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206481+0000","last_change":"2023-07-14T19:02:30.572052+0000","last_active":"2023-07-14T19:03:30.206481+0000","last_peered":"2023-07-14T19:03:30.206481+0000","last_clean":"2023-07-14T19:03:30.206481+0000","last_became_active":"2023-07-14T19:02:30.571739+0000","last_became_peered":"2023-07-14T19:02:30.571739+0000","last_unstale":"2023-07-14T19:03:30.206481+0000","last_undegraded":"2023-07-14T19:03:30.206481+0000","last_fullsized":"2023-07-14T19:03:30.206481+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:27:03.443418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219888+0000","last_change":"2023-07-14T19:02:34.576663+0000","last_active":"2023-07-14T19:03:31.219888+0000","last_peered":"2023-07-14T19:03:31.219888+0000","last_clean":"2023-07-14T19:03:31.219888+0000","last_became_active":"2023-07-14T19:02:34.576162+0000","last_became_peered":"2023-07-14T19:02:34.576162+0000","last_unstale":"2023-07-14T19:03:31.219888+0000","last_undegraded":"2023-07-14T19:03:31.219888+0000","last_fullsized":"2023-07-14T19:03:31.219888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:50.099226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":2,"reported_epoch":86,"state":"peering","last_fresh":"2023-07-14T19:02:50.616451+0000","last_change":"2023-07-14T19:03:30.725628+0000","last_active":"2023-07-14T19:03:30.725628+0000","last_peered":"2023-07-14T19:02:50.616451+0000","last_clean":"2023-07-14T19:02:50.616451+0000","last_became_active":"2023-07-14T19:02:29.938630+0000","last_became_peered":"2023-07-14T19:02:29.938630+0000","last_unstale":"2023-07-14T19:02:50.616451+0000","last_undegraded":"2023-07-14T19:02:50.616451+0000","last_fullsized":"2023-07-14T19:02:50.616451+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608493+0000","last_change":"2023-07-14T19:02:30.573798+0000","last_active":"2023-07-14T19:02:35.608493+0000","last_peered":"2023-07-14T19:02:35.608493+0000","last_clean":"2023-07-14T19:02:35.608493+0000","last_became_active":"2023-07-14T19:02:30.572568+0000","last_became_peered":"2023-07-14T19:02:30.572568+0000","last_unstale":"2023-07-14T19:02:35.608493+0000","last_undegraded":"2023-07-14T19:02:35.608493+0000","last_fullsized":"2023-07-14T19:02:35.608493+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:36:02.901707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208768+0000","last_change":"2023-07-14T19:02:34.582256+0000","last_active":"2023-07-14T19:03:30.208768+0000","last_peered":"2023-07-14T19:03:30.208768+0000","last_clean":"2023-07-14T19:03:30.208768+0000","last_became_active":"2023-07-14T19:02:34.581799+0000","last_became_peered":"2023-07-14T19:02:34.581799+0000","last_unstale":"2023-07-14T19:03:30.208768+0000","last_undegraded":"2023-07-14T19:03:30.208768+0000","last_fullsized":"2023-07-14T19:03:30.208768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:12.667561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219545+0000","last_change":"2023-07-14T19:02:29.943874+0000","last_active":"2023-07-14T19:03:31.219545+0000","last_peered":"2023-07-14T19:03:31.219545+0000","last_clean":"2023-07-14T19:03:31.219545+0000","last_became_active":"2023-07-14T19:02:29.943596+0000","last_became_peered":"2023-07-14T19:02:29.943596+0000","last_unstale":"2023-07-14T19:03:31.219545+0000","last_undegraded":"2023-07-14T19:03:31.219545+0000","last_fullsized":"2023-07-14T19:03:31.219545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:17.878050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214284+0000","last_change":"2023-07-14T19:02:30.572157+0000","last_active":"2023-07-14T19:03:30.214284+0000","last_peered":"2023-07-14T19:03:30.214284+0000","last_clean":"2023-07-14T19:03:30.214284+0000","last_became_active":"2023-07-14T19:02:30.571877+0000","last_became_peered":"2023-07-14T19:02:30.571877+0000","last_unstale":"2023-07-14T19:03:30.214284+0000","last_undegraded":"2023-07-14T19:03:30.214284+0000","last_fullsized":"2023-07-14T19:03:30.214284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:15:54.226221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210374+0000","last_change":"2023-07-14T19:02:34.588594+0000","last_active":"2023-07-14T19:03:31.210374+0000","last_peered":"2023-07-14T19:03:31.210374+0000","last_clean":"2023-07-14T19:03:31.210374+0000","last_became_active":"2023-07-14T19:02:34.587763+0000","last_became_peered":"2023-07-14T19:02:34.587763+0000","last_unstale":"2023-07-14T19:03:31.210374+0000","last_undegraded":"2023-07-14T19:03:31.210374+0000","last_fullsized":"2023-07-14T19:03:31.210374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:02.399855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212239+0000","last_change":"2023-07-14T19:02:29.939145+0000","last_active":"2023-07-14T19:03:31.212239+0000","last_peered":"2023-07-14T19:03:31.212239+0000","last_clean":"2023-07-14T19:03:31.212239+0000","last_became_active":"2023-07-14T19:02:29.938816+0000","last_became_peered":"2023-07-14T19:02:29.938816+0000","last_unstale":"2023-07-14T19:03:31.212239+0000","last_undegraded":"2023-07-14T19:03:31.212239+0000","last_fullsized":"2023-07-14T19:03:31.212239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:05.245627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.213928+0000","last_change":"2023-07-14T19:02:30.572694+0000","last_active":"2023-07-14T19:03:30.213928+0000","last_peered":"2023-07-14T19:03:30.213928+0000","last_clean":"2023-07-14T19:03:30.213928+0000","last_became_active":"2023-07-14T19:02:30.572391+0000","last_became_peered":"2023-07-14T19:02:30.572391+0000","last_unstale":"2023-07-14T19:03:30.213928+0000","last_undegraded":"2023-07-14T19:03:30.213928+0000","last_fullsized":"2023-07-14T19:03:30.213928+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:44:21.348702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609130+0000","last_change":"2023-07-14T19:02:34.582476+0000","last_active":"2023-07-14T19:02:35.609130+0000","last_peered":"2023-07-14T19:02:35.609130+0000","last_clean":"2023-07-14T19:02:35.609130+0000","last_became_active":"2023-07-14T19:02:34.582083+0000","last_became_peered":"2023-07-14T19:02:34.582083+0000","last_unstale":"2023-07-14T19:02:35.609130+0000","last_undegraded":"2023-07-14T19:02:35.609130+0000","last_fullsized":"2023-07-14T19:02:35.609130+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:43:53.732894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":2,"reported_epoch":86,"state":"peering","last_fresh":"2023-07-14T19:02:49.197385+0000","last_change":"2023-07-14T19:03:30.725597+0000","last_active":"2023-07-14T19:03:30.725597+0000","last_peered":"2023-07-14T19:02:49.197385+0000","last_clean":"2023-07-14T19:02:49.197385+0000","last_became_active":"2023-07-14T19:02:29.941103+0000","last_became_peered":"2023-07-14T19:02:29.941103+0000","last_unstale":"2023-07-14T19:02:49.197385+0000","last_undegraded":"2023-07-14T19:02:49.197385+0000","last_fullsized":"2023-07-14T19:02:49.197385+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.282151+0000","last_change":"2023-07-14T19:03:31.282151+0000","last_active":"2023-07-14T19:03:31.282151+0000","last_peered":"2023-07-14T19:03:31.282151+0000","last_clean":"2023-07-14T19:03:31.282151+0000","last_became_active":"2023-07-14T19:03:31.281763+0000","last_became_peered":"2023-07-14T19:03:31.281763+0000","last_unstale":"2023-07-14T19:03:31.282151+0000","last_undegraded":"2023-07-14T19:03:31.282151+0000","last_fullsized":"2023-07-14T19:03:31.282151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:13.429632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218880+0000","last_change":"2023-07-14T19:02:34.584170+0000","last_active":"2023-07-14T19:03:31.218880+0000","last_peered":"2023-07-14T19:03:31.218880+0000","last_clean":"2023-07-14T19:03:31.218880+0000","last_became_active":"2023-07-14T19:02:34.583773+0000","last_became_peered":"2023-07-14T19:02:34.583773+0000","last_unstale":"2023-07-14T19:03:31.218880+0000","last_undegraded":"2023-07-14T19:03:31.218880+0000","last_fullsized":"2023-07-14T19:03:31.218880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:32:31.049038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.207509+0000","last_change":"2023-07-14T19:02:29.938386+0000","last_active":"2023-07-14T19:03:30.207509+0000","last_peered":"2023-07-14T19:03:30.207509+0000","last_clean":"2023-07-14T19:03:30.207509+0000","last_became_active":"2023-07-14T19:02:29.937394+0000","last_became_peered":"2023-07-14T19:02:29.937394+0000","last_unstale":"2023-07-14T19:03:30.207509+0000","last_undegraded":"2023-07-14T19:03:30.207509+0000","last_fullsized":"2023-07-14T19:03:30.207509+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:17:58.170429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208774+0000","last_change":"2023-07-14T19:02:30.572485+0000","last_active":"2023-07-14T19:03:30.208774+0000","last_peered":"2023-07-14T19:03:30.208774+0000","last_clean":"2023-07-14T19:03:30.208774+0000","last_became_active":"2023-07-14T19:02:30.572177+0000","last_became_peered":"2023-07-14T19:02:30.572177+0000","last_unstale":"2023-07-14T19:03:30.208774+0000","last_undegraded":"2023-07-14T19:03:30.208774+0000","last_fullsized":"2023-07-14T19:03:30.208774+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:56:12.967413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.207256+0000","last_change":"2023-07-14T19:02:34.582340+0000","last_active":"2023-07-14T19:03:30.207256+0000","last_peered":"2023-07-14T19:03:30.207256+0000","last_clean":"2023-07-14T19:03:30.207256+0000","last_became_active":"2023-07-14T19:02:34.581936+0000","last_became_peered":"2023-07-14T19:02:34.581936+0000","last_unstale":"2023-07-14T19:03:30.207256+0000","last_undegraded":"2023-07-14T19:03:30.207256+0000","last_fullsized":"2023-07-14T19:03:30.207256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:31.710972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210852+0000","last_change":"2023-07-14T19:02:29.941404+0000","last_active":"2023-07-14T19:03:31.210852+0000","last_peered":"2023-07-14T19:03:31.210852+0000","last_clean":"2023-07-14T19:03:31.210852+0000","last_became_active":"2023-07-14T19:02:29.941052+0000","last_became_peered":"2023-07-14T19:02:29.941052+0000","last_unstale":"2023-07-14T19:03:31.210852+0000","last_undegraded":"2023-07-14T19:03:31.210852+0000","last_fullsized":"2023-07-14T19:03:31.210852+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:23:27.513794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214237+0000","last_change":"2023-07-14T19:02:30.565671+0000","last_active":"2023-07-14T19:03:31.214237+0000","last_peered":"2023-07-14T19:03:31.214237+0000","last_clean":"2023-07-14T19:03:31.214237+0000","last_became_active":"2023-07-14T19:02:30.565375+0000","last_became_peered":"2023-07-14T19:02:30.565375+0000","last_unstale":"2023-07-14T19:03:31.214237+0000","last_undegraded":"2023-07-14T19:03:31.214237+0000","last_fullsized":"2023-07-14T19:03:31.214237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:12:48.571954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608815+0000","last_change":"2023-07-14T19:02:34.577692+0000","last_active":"2023-07-14T19:02:35.608815+0000","last_peered":"2023-07-14T19:02:35.608815+0000","last_clean":"2023-07-14T19:02:35.608815+0000","last_became_active":"2023-07-14T19:02:34.577329+0000","last_became_peered":"2023-07-14T19:02:34.577329+0000","last_unstale":"2023-07-14T19:02:35.608815+0000","last_undegraded":"2023-07-14T19:02:35.608815+0000","last_fullsized":"2023-07-14T19:02:35.608815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:25:42.179550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214219+0000","last_change":"2023-07-14T19:02:29.938330+0000","last_active":"2023-07-14T19:03:30.214219+0000","last_peered":"2023-07-14T19:03:30.214219+0000","last_clean":"2023-07-14T19:03:30.214219+0000","last_became_active":"2023-07-14T19:02:29.938022+0000","last_became_peered":"2023-07-14T19:02:29.938022+0000","last_unstale":"2023-07-14T19:03:30.214219+0000","last_undegraded":"2023-07-14T19:03:30.214219+0000","last_fullsized":"2023-07-14T19:03:30.214219+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:53.432786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213923+0000","last_change":"2023-07-14T19:02:30.569353+0000","last_active":"2023-07-14T19:03:31.213923+0000","last_peered":"2023-07-14T19:03:31.213923+0000","last_clean":"2023-07-14T19:03:31.213923+0000","last_became_active":"2023-07-14T19:02:30.569037+0000","last_became_peered":"2023-07-14T19:02:30.569037+0000","last_unstale":"2023-07-14T19:03:31.213923+0000","last_undegraded":"2023-07-14T19:03:31.213923+0000","last_fullsized":"2023-07-14T19:03:31.213923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:38.723738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609120+0000","last_change":"2023-07-14T19:02:34.580786+0000","last_active":"2023-07-14T19:02:35.609120+0000","last_peered":"2023-07-14T19:02:35.609120+0000","last_clean":"2023-07-14T19:02:35.609120+0000","last_became_active":"2023-07-14T19:02:34.580395+0000","last_became_peered":"2023-07-14T19:02:34.580395+0000","last_unstale":"2023-07-14T19:02:35.609120+0000","last_undegraded":"2023-07-14T19:02:35.609120+0000","last_fullsized":"2023-07-14T19:02:35.609120+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:12:24.810182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218850+0000","last_change":"2023-07-14T19:02:30.574302+0000","last_active":"2023-07-14T19:03:31.218850+0000","last_peered":"2023-07-14T19:03:31.218850+0000","last_clean":"2023-07-14T19:03:31.218850+0000","last_became_active":"2023-07-14T19:02:30.573912+0000","last_became_peered":"2023-07-14T19:02:30.573912+0000","last_unstale":"2023-07-14T19:03:31.218850+0000","last_undegraded":"2023-07-14T19:03:31.218850+0000","last_fullsized":"2023-07-14T19:03:31.218850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:57.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":66,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:03:28.673439+0000","last_change":"2023-07-14T19:02:29.942302+0000","last_active":"2023-07-14T19:03:28.673439+0000","last_peered":"2023-07-14T19:03:28.673439+0000","last_clean":"2023-07-14T19:03:28.673439+0000","last_became_active":"2023-07-14T19:02:29.942005+0000","last_became_peered":"2023-07-14T19:02:29.942005+0000","last_unstale":"2023-07-14T19:03:28.673439+0000","last_undegraded":"2023-07-14T19:03:28.673439+0000","last_fullsized":"2023-07-14T19:03:28.673439+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:47:55.061870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":40,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214474+0000","last_change":"2023-07-14T19:02:34.578248+0000","last_active":"2023-07-14T19:03:30.214474+0000","last_peered":"2023-07-14T19:03:30.214474+0000","last_clean":"2023-07-14T19:03:30.214474+0000","last_became_active":"2023-07-14T19:02:34.577930+0000","last_became_peered":"2023-07-14T19:02:34.577930+0000","last_unstale":"2023-07-14T19:03:30.214474+0000","last_undegraded":"2023-07-14T19:03:30.214474+0000","last_fullsized":"2023-07-14T19:03:30.214474+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:40:19.412781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811402+0000","last_change":"2023-07-14T19:02:30.582892+0000","last_active":"2023-07-14T19:02:35.811402+0000","last_peered":"2023-07-14T19:02:35.811402+0000","last_clean":"2023-07-14T19:02:35.811402+0000","last_became_active":"2023-07-14T19:02:30.582588+0000","last_became_peered":"2023-07-14T19:02:30.582588+0000","last_unstale":"2023-07-14T19:02:35.811402+0000","last_undegraded":"2023-07-14T19:02:35.811402+0000","last_fullsized":"2023-07-14T19:02:35.811402+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:00:14.845398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214203+0000","last_change":"2023-07-14T19:02:29.935857+0000","last_active":"2023-07-14T19:03:31.214203+0000","last_peered":"2023-07-14T19:03:31.214203+0000","last_clean":"2023-07-14T19:03:31.214203+0000","last_became_active":"2023-07-14T19:02:29.935489+0000","last_became_peered":"2023-07-14T19:02:29.935489+0000","last_unstale":"2023-07-14T19:03:31.214203+0000","last_undegraded":"2023-07-14T19:03:31.214203+0000","last_fullsized":"2023-07-14T19:03:31.214203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:08.956235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":338,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.213796+0000","last_change":"2023-07-14T19:01:31.967476+0000","last_active":"2023-07-14T19:03:30.213796+0000","last_peered":"2023-07-14T19:03:30.213796+0000","last_clean":"2023-07-14T19:03:30.213796+0000","last_became_active":"2023-07-14T19:01:31.967146+0000","last_became_peered":"2023-07-14T19:01:31.967146+0000","last_unstale":"2023-07-14T19:03:30.213796+0000","last_undegraded":"2023-07-14T19:03:30.213796+0000","last_fullsized":"2023-07-14T19:03:30.213796+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_clean_scrub_stamp":"2023-07-14T18:58:40.634620+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:41:56.581484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219439+0000","last_change":"2023-07-14T19:02:34.571823+0000","last_active":"2023-07-14T19:03:31.219439+0000","last_peered":"2023-07-14T19:03:31.219439+0000","last_clean":"2023-07-14T19:03:31.219439+0000","last_became_active":"2023-07-14T19:02:34.571359+0000","last_became_peered":"2023-07-14T19:02:34.571359+0000","last_unstale":"2023-07-14T19:03:31.219439+0000","last_undegraded":"2023-07-14T19:03:31.219439+0000","last_fullsized":"2023-07-14T19:03:31.219439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:28.423706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608304+0000","last_change":"2023-07-14T19:02:30.573736+0000","last_active":"2023-07-14T19:02:35.608304+0000","last_peered":"2023-07-14T19:02:35.608304+0000","last_clean":"2023-07-14T19:02:35.608304+0000","last_became_active":"2023-07-14T19:02:30.573381+0000","last_became_peered":"2023-07-14T19:02:30.573381+0000","last_unstale":"2023-07-14T19:02:35.608304+0000","last_undegraded":"2023-07-14T19:02:35.608304+0000","last_fullsized":"2023-07-14T19:02:35.608304+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:21:36.599123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.213908+0000","last_change":"2023-07-14T19:02:29.937660+0000","last_active":"2023-07-14T19:03:30.213908+0000","last_peered":"2023-07-14T19:03:30.213908+0000","last_clean":"2023-07-14T19:03:30.213908+0000","last_became_active":"2023-07-14T19:02:29.937348+0000","last_became_peered":"2023-07-14T19:02:29.937348+0000","last_unstale":"2023-07-14T19:03:30.213908+0000","last_undegraded":"2023-07-14T19:03:30.213908+0000","last_fullsized":"2023-07-14T19:03:30.213908+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:11:25.564711+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.209687+0000","last_change":"2023-07-14T19:02:34.581534+0000","last_active":"2023-07-14T19:03:30.209687+0000","last_peered":"2023-07-14T19:03:30.209687+0000","last_clean":"2023-07-14T19:03:30.209687+0000","last_became_active":"2023-07-14T19:02:34.580998+0000","last_became_peered":"2023-07-14T19:02:34.580998+0000","last_unstale":"2023-07-14T19:03:30.209687+0000","last_undegraded":"2023-07-14T19:03:30.209687+0000","last_fullsized":"2023-07-14T19:03:30.209687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:33:51.597227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206926+0000","last_change":"2023-07-14T19:02:29.942336+0000","last_active":"2023-07-14T19:03:30.206926+0000","last_peered":"2023-07-14T19:03:30.206926+0000","last_clean":"2023-07-14T19:03:30.206926+0000","last_became_active":"2023-07-14T19:02:29.942033+0000","last_became_peered":"2023-07-14T19:02:29.942033+0000","last_unstale":"2023-07-14T19:03:30.206926+0000","last_undegraded":"2023-07-14T19:03:30.206926+0000","last_fullsized":"2023-07-14T19:03:30.206926+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:40:32.682171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608984+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:02:35.608984+0000","last_peered":"2023-07-14T19:02:35.608984+0000","last_clean":"2023-07-14T19:02:35.608984+0000","last_became_active":"2023-07-14T19:02:30.571129+0000","last_became_peered":"2023-07-14T19:02:30.571129+0000","last_unstale":"2023-07-14T19:02:35.608984+0000","last_undegraded":"2023-07-14T19:02:35.608984+0000","last_fullsized":"2023-07-14T19:02:35.608984+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:12:17.668104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811239+0000","last_change":"2023-07-14T19:02:34.580740+0000","last_active":"2023-07-14T19:02:35.811239+0000","last_peered":"2023-07-14T19:02:35.811239+0000","last_clean":"2023-07-14T19:02:35.811239+0000","last_became_active":"2023-07-14T19:02:34.580344+0000","last_became_peered":"2023-07-14T19:02:34.580344+0000","last_unstale":"2023-07-14T19:02:35.811239+0000","last_undegraded":"2023-07-14T19:02:35.811239+0000","last_fullsized":"2023-07-14T19:02:35.811239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:26:55.627423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'5","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210352+0000","last_change":"2023-07-14T19:02:29.941349+0000","last_active":"2023-07-14T19:03:31.210352+0000","last_peered":"2023-07-14T19:03:31.210352+0000","last_clean":"2023-07-14T19:03:31.210352+0000","last_became_active":"2023-07-14T19:02:29.941039+0000","last_became_peered":"2023-07-14T19:02:29.941039+0000","last_unstale":"2023-07-14T19:03:31.210352+0000","last_undegraded":"2023-07-14T19:03:31.210352+0000","last_fullsized":"2023-07-14T19:03:31.210352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:38.367807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811807+0000","last_change":"2023-07-14T19:02:30.577546+0000","last_active":"2023-07-14T19:02:35.811807+0000","last_peered":"2023-07-14T19:02:35.811807+0000","last_clean":"2023-07-14T19:02:35.811807+0000","last_became_active":"2023-07-14T19:02:30.576833+0000","last_became_peered":"2023-07-14T19:02:30.576833+0000","last_unstale":"2023-07-14T19:02:35.811807+0000","last_undegraded":"2023-07-14T19:02:35.811807+0000","last_fullsized":"2023-07-14T19:02:35.811807+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:06:47.060524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":13,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608552+0000","last_change":"2023-07-14T19:02:34.580655+0000","last_active":"2023-07-14T19:02:35.608552+0000","last_peered":"2023-07-14T19:02:35.608552+0000","last_clean":"2023-07-14T19:02:35.608552+0000","last_became_active":"2023-07-14T19:02:34.580186+0000","last_became_peered":"2023-07-14T19:02:34.580186+0000","last_unstale":"2023-07-14T19:02:35.608552+0000","last_undegraded":"2023-07-14T19:02:35.608552+0000","last_fullsized":"2023-07-14T19:02:35.608552+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:30.211525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212620+0000","last_change":"2023-07-14T19:02:29.938343+0000","last_active":"2023-07-14T19:03:31.212620+0000","last_peered":"2023-07-14T19:03:31.212620+0000","last_clean":"2023-07-14T19:03:31.212620+0000","last_became_active":"2023-07-14T19:02:29.938053+0000","last_became_peered":"2023-07-14T19:02:29.938053+0000","last_unstale":"2023-07-14T19:03:31.212620+0000","last_undegraded":"2023-07-14T19:03:31.212620+0000","last_fullsized":"2023-07-14T19:03:31.212620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:50:45.655040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208419+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:03:31.208419+0000","last_peered":"2023-07-14T19:03:31.208419+0000","last_clean":"2023-07-14T19:03:31.208419+0000","last_became_active":"2023-07-14T19:02:30.572436+0000","last_became_peered":"2023-07-14T19:02:30.572436+0000","last_unstale":"2023-07-14T19:03:31.208419+0000","last_undegraded":"2023-07-14T19:03:31.208419+0000","last_fullsized":"2023-07-14T19:03:31.208419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:11:10.005214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213237+0000","last_change":"2023-07-14T19:02:34.582900+0000","last_active":"2023-07-14T19:03:31.213237+0000","last_peered":"2023-07-14T19:03:31.213237+0000","last_clean":"2023-07-14T19:03:31.213237+0000","last_became_active":"2023-07-14T19:02:34.582493+0000","last_became_peered":"2023-07-14T19:02:34.582493+0000","last_unstale":"2023-07-14T19:03:31.213237+0000","last_undegraded":"2023-07-14T19:03:31.213237+0000","last_fullsized":"2023-07-14T19:03:31.213237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:00:45.030002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.280975+0000","last_change":"2023-07-14T19:02:29.941819+0000","last_active":"2023-07-14T19:03:31.280975+0000","last_peered":"2023-07-14T19:03:31.280975+0000","last_clean":"2023-07-14T19:03:31.280975+0000","last_became_active":"2023-07-14T19:02:29.941469+0000","last_became_peered":"2023-07-14T19:02:29.941469+0000","last_unstale":"2023-07-14T19:03:31.280975+0000","last_undegraded":"2023-07-14T19:03:31.280975+0000","last_fullsized":"2023-07-14T19:03:31.280975+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:28.479332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206177+0000","last_change":"2023-07-14T19:02:30.565125+0000","last_active":"2023-07-14T19:03:30.206177+0000","last_peered":"2023-07-14T19:03:30.206177+0000","last_clean":"2023-07-14T19:03:30.206177+0000","last_became_active":"2023-07-14T19:02:30.564820+0000","last_became_peered":"2023-07-14T19:02:30.564820+0000","last_unstale":"2023-07-14T19:03:30.206177+0000","last_undegraded":"2023-07-14T19:03:30.206177+0000","last_fullsized":"2023-07-14T19:03:30.206177+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:54:35.924491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.221013+0000","last_change":"2023-07-14T19:03:31.220582+0000","last_active":"2023-07-14T19:03:31.221013+0000","last_peered":"2023-07-14T19:03:31.221013+0000","last_clean":"2023-07-14T19:03:31.221013+0000","last_became_active":"2023-07-14T19:03:31.220162+0000","last_became_peered":"2023-07-14T19:03:31.220162+0000","last_unstale":"2023-07-14T19:03:31.221013+0000","last_undegraded":"2023-07-14T19:03:31.221013+0000","last_fullsized":"2023-07-14T19:03:31.221013+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:00.068361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":25,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:49.195784+0000","last_change":"2023-07-14T19:02:29.938027+0000","last_active":"2023-07-14T19:02:49.195784+0000","last_peered":"2023-07-14T19:02:49.195784+0000","last_clean":"2023-07-14T19:02:49.195784+0000","last_became_active":"2023-07-14T19:02:29.937690+0000","last_became_peered":"2023-07-14T19:02:29.937690+0000","last_unstale":"2023-07-14T19:02:49.195784+0000","last_undegraded":"2023-07-14T19:02:49.195784+0000","last_fullsized":"2023-07-14T19:02:49.195784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:27:03.527958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.207888+0000","last_change":"2023-07-14T19:02:30.576369+0000","last_active":"2023-07-14T19:03:30.207888+0000","last_peered":"2023-07-14T19:03:30.207888+0000","last_clean":"2023-07-14T19:03:30.207888+0000","last_became_active":"2023-07-14T19:02:30.576113+0000","last_became_peered":"2023-07-14T19:02:30.576113+0000","last_unstale":"2023-07-14T19:03:30.207888+0000","last_undegraded":"2023-07-14T19:03:30.207888+0000","last_fullsized":"2023-07-14T19:03:30.207888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:03:26.824693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213432+0000","last_change":"2023-07-14T19:02:34.583724+0000","last_active":"2023-07-14T19:03:31.213432+0000","last_peered":"2023-07-14T19:03:31.213432+0000","last_clean":"2023-07-14T19:03:31.213432+0000","last_became_active":"2023-07-14T19:02:34.583218+0000","last_became_peered":"2023-07-14T19:02:34.583218+0000","last_unstale":"2023-07-14T19:03:31.213432+0000","last_undegraded":"2023-07-14T19:03:31.213432+0000","last_fullsized":"2023-07-14T19:03:31.213432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:28:38.807293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608853+0000","last_change":"2023-07-14T19:02:29.938908+0000","last_active":"2023-07-14T19:02:35.608853+0000","last_peered":"2023-07-14T19:02:35.608853+0000","last_clean":"2023-07-14T19:02:35.608853+0000","last_became_active":"2023-07-14T19:02:29.938650+0000","last_became_peered":"2023-07-14T19:02:29.938650+0000","last_unstale":"2023-07-14T19:02:35.608853+0000","last_undegraded":"2023-07-14T19:02:35.608853+0000","last_fullsized":"2023-07-14T19:02:35.608853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:19.121855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811550+0000","last_change":"2023-07-14T19:02:30.571870+0000","last_active":"2023-07-14T19:02:35.811550+0000","last_peered":"2023-07-14T19:02:35.811550+0000","last_clean":"2023-07-14T19:02:35.811550+0000","last_became_active":"2023-07-14T19:02:30.571521+0000","last_became_peered":"2023-07-14T19:02:30.571521+0000","last_unstale":"2023-07-14T19:02:35.811550+0000","last_undegraded":"2023-07-14T19:02:35.811550+0000","last_fullsized":"2023-07-14T19:02:35.811550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:43:37.931915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213270+0000","last_change":"2023-07-14T19:02:34.589900+0000","last_active":"2023-07-14T19:03:31.213270+0000","last_peered":"2023-07-14T19:03:31.213270+0000","last_clean":"2023-07-14T19:03:31.213270+0000","last_became_active":"2023-07-14T19:02:34.589481+0000","last_became_peered":"2023-07-14T19:02:34.589481+0000","last_unstale":"2023-07-14T19:03:31.213270+0000","last_undegraded":"2023-07-14T19:03:31.213270+0000","last_fullsized":"2023-07-14T19:03:31.213270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:07.853098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":29,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.206814+0000","last_change":"2023-07-14T19:02:29.940315+0000","last_active":"2023-07-14T19:03:30.206814+0000","last_peered":"2023-07-14T19:03:30.206814+0000","last_clean":"2023-07-14T19:03:30.206814+0000","last_became_active":"2023-07-14T19:02:29.939991+0000","last_became_peered":"2023-07-14T19:02:29.939991+0000","last_unstale":"2023-07-14T19:03:30.206814+0000","last_undegraded":"2023-07-14T19:03:30.206814+0000","last_fullsized":"2023-07-14T19:03:30.206814+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:55:40.497517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214331+0000","last_change":"2023-07-14T19:02:30.583959+0000","last_active":"2023-07-14T19:03:30.214331+0000","last_peered":"2023-07-14T19:03:30.214331+0000","last_clean":"2023-07-14T19:03:30.214331+0000","last_became_active":"2023-07-14T19:02:30.583657+0000","last_became_peered":"2023-07-14T19:02:30.583657+0000","last_unstale":"2023-07-14T19:03:30.214331+0000","last_undegraded":"2023-07-14T19:03:30.214331+0000","last_fullsized":"2023-07-14T19:03:30.214331+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:38:23.526665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281103+0000","last_change":"2023-07-14T19:02:34.579328+0000","last_active":"2023-07-14T19:03:31.281103+0000","last_peered":"2023-07-14T19:03:31.281103+0000","last_clean":"2023-07-14T19:03:31.281103+0000","last_became_active":"2023-07-14T19:02:34.578904+0000","last_became_peered":"2023-07-14T19:02:34.578904+0000","last_unstale":"2023-07-14T19:03:31.281103+0000","last_undegraded":"2023-07-14T19:03:31.281103+0000","last_fullsized":"2023-07-14T19:03:31.281103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:15.218624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":26,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:50.207810+0000","last_change":"2023-07-14T19:02:29.940972+0000","last_active":"2023-07-14T19:02:50.207810+0000","last_peered":"2023-07-14T19:02:50.207810+0000","last_clean":"2023-07-14T19:02:50.207810+0000","last_became_active":"2023-07-14T19:02:29.940670+0000","last_became_peered":"2023-07-14T19:02:29.940670+0000","last_unstale":"2023-07-14T19:02:50.207810+0000","last_undegraded":"2023-07-14T19:02:50.207810+0000","last_fullsized":"2023-07-14T19:02:50.207810+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:41:09.182828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214674+0000","last_change":"2023-07-14T19:02:30.583714+0000","last_active":"2023-07-14T19:03:30.214674+0000","last_peered":"2023-07-14T19:03:30.214674+0000","last_clean":"2023-07-14T19:03:30.214674+0000","last_became_active":"2023-07-14T19:02:30.583404+0000","last_became_peered":"2023-07-14T19:02:30.583404+0000","last_unstale":"2023-07-14T19:03:30.214674+0000","last_undegraded":"2023-07-14T19:03:30.214674+0000","last_fullsized":"2023-07-14T19:03:30.214674+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:39:26.680582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207943+0000","last_change":"2023-07-14T19:02:34.582985+0000","last_active":"2023-07-14T19:03:31.207943+0000","last_peered":"2023-07-14T19:03:31.207943+0000","last_clean":"2023-07-14T19:03:31.207943+0000","last_became_active":"2023-07-14T19:02:34.582606+0000","last_became_peered":"2023-07-14T19:02:34.582606+0000","last_unstale":"2023-07-14T19:03:31.207943+0000","last_undegraded":"2023-07-14T19:03:31.207943+0000","last_fullsized":"2023-07-14T19:03:31.207943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:55:50.958196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219531+0000","last_change":"2023-07-14T19:02:29.944064+0000","last_active":"2023-07-14T19:03:31.219531+0000","last_peered":"2023-07-14T19:03:31.219531+0000","last_clean":"2023-07-14T19:03:31.219531+0000","last_became_active":"2023-07-14T19:02:29.943770+0000","last_became_peered":"2023-07-14T19:02:29.943770+0000","last_unstale":"2023-07-14T19:03:31.219531+0000","last_undegraded":"2023-07-14T19:03:31.219531+0000","last_fullsized":"2023-07-14T19:03:31.219531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:02:39.670163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214408+0000","last_change":"2023-07-14T19:02:30.580668+0000","last_active":"2023-07-14T19:03:30.214408+0000","last_peered":"2023-07-14T19:03:30.214408+0000","last_clean":"2023-07-14T19:03:30.214408+0000","last_became_active":"2023-07-14T19:02:30.580297+0000","last_became_peered":"2023-07-14T19:02:30.580297+0000","last_unstale":"2023-07-14T19:03:30.214408+0000","last_undegraded":"2023-07-14T19:03:30.214408+0000","last_fullsized":"2023-07-14T19:03:30.214408+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:37:20.057488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.217879+0000","last_change":"2023-07-14T19:02:34.578894+0000","last_active":"2023-07-14T19:03:31.217879+0000","last_peered":"2023-07-14T19:03:31.217879+0000","last_clean":"2023-07-14T19:03:31.217879+0000","last_became_active":"2023-07-14T19:02:34.578463+0000","last_became_peered":"2023-07-14T19:02:34.578463+0000","last_unstale":"2023-07-14T19:03:31.217879+0000","last_undegraded":"2023-07-14T19:03:31.217879+0000","last_fullsized":"2023-07-14T19:03:31.217879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:53.046945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609171+0000","last_change":"2023-07-14T19:02:29.938430+0000","last_active":"2023-07-14T19:02:35.609171+0000","last_peered":"2023-07-14T19:02:35.609171+0000","last_clean":"2023-07-14T19:02:35.609171+0000","last_became_active":"2023-07-14T19:02:29.938118+0000","last_became_peered":"2023-07-14T19:02:29.938118+0000","last_unstale":"2023-07-14T19:02:35.609171+0000","last_undegraded":"2023-07-14T19:02:35.609171+0000","last_fullsized":"2023-07-14T19:02:35.609171+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:57:08.670837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.214695+0000","last_change":"2023-07-14T19:02:30.575124+0000","last_active":"2023-07-14T19:03:30.214695+0000","last_peered":"2023-07-14T19:03:30.214695+0000","last_clean":"2023-07-14T19:03:30.214695+0000","last_became_active":"2023-07-14T19:02:30.574805+0000","last_became_peered":"2023-07-14T19:02:30.574805+0000","last_unstale":"2023-07-14T19:03:30.214695+0000","last_undegraded":"2023-07-14T19:03:30.214695+0000","last_fullsized":"2023-07-14T19:03:30.214695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:22:41.478433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208131+0000","last_change":"2023-07-14T19:02:30.573897+0000","last_active":"2023-07-14T19:03:30.208131+0000","last_peered":"2023-07-14T19:03:30.208131+0000","last_clean":"2023-07-14T19:03:30.208131+0000","last_became_active":"2023-07-14T19:02:30.573585+0000","last_became_peered":"2023-07-14T19:02:30.573585+0000","last_unstale":"2023-07-14T19:03:30.208131+0000","last_undegraded":"2023-07-14T19:03:30.208131+0000","last_fullsized":"2023-07-14T19:03:30.208131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:32:55.646800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608862+0000","last_change":"2023-07-14T19:02:30.572390+0000","last_active":"2023-07-14T19:02:35.608862+0000","last_peered":"2023-07-14T19:02:35.608862+0000","last_clean":"2023-07-14T19:02:35.608862+0000","last_became_active":"2023-07-14T19:02:30.572093+0000","last_became_peered":"2023-07-14T19:02:30.572093+0000","last_unstale":"2023-07-14T19:02:35.608862+0000","last_undegraded":"2023-07-14T19:02:35.608862+0000","last_fullsized":"2023-07-14T19:02:35.608862+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:55:53.445808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608706+0000","last_change":"2023-07-14T19:02:30.572548+0000","last_active":"2023-07-14T19:02:35.608706+0000","last_peered":"2023-07-14T19:02:35.608706+0000","last_clean":"2023-07-14T19:02:35.608706+0000","last_became_active":"2023-07-14T19:02:30.572260+0000","last_became_peered":"2023-07-14T19:02:30.572260+0000","last_unstale":"2023-07-14T19:02:35.608706+0000","last_undegraded":"2023-07-14T19:02:35.608706+0000","last_fullsized":"2023-07-14T19:02:35.608706+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:43:10.013952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208138+0000","last_change":"2023-07-14T19:02:30.569856+0000","last_active":"2023-07-14T19:03:31.208138+0000","last_peered":"2023-07-14T19:03:31.208138+0000","last_clean":"2023-07-14T19:03:31.208138+0000","last_became_active":"2023-07-14T19:02:30.569546+0000","last_became_peered":"2023-07-14T19:02:30.569546+0000","last_unstale":"2023-07-14T19:03:31.208138+0000","last_undegraded":"2023-07-14T19:03:31.208138+0000","last_fullsized":"2023-07-14T19:03:31.208138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:43.152189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608964+0000","last_change":"2023-07-14T19:02:30.570836+0000","last_active":"2023-07-14T19:02:35.608964+0000","last_peered":"2023-07-14T19:02:35.608964+0000","last_clean":"2023-07-14T19:02:35.608964+0000","last_became_active":"2023-07-14T19:02:30.569675+0000","last_became_peered":"2023-07-14T19:02:30.569675+0000","last_unstale":"2023-07-14T19:02:35.608964+0000","last_undegraded":"2023-07-14T19:02:35.608964+0000","last_fullsized":"2023-07-14T19:02:35.608964+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:50.028159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.209264+0000","last_change":"2023-07-14T19:02:30.566846+0000","last_active":"2023-07-14T19:03:30.209264+0000","last_peered":"2023-07-14T19:03:30.209264+0000","last_clean":"2023-07-14T19:03:30.209264+0000","last_became_active":"2023-07-14T19:02:30.566525+0000","last_became_peered":"2023-07-14T19:02:30.566525+0000","last_unstale":"2023-07-14T19:03:30.209264+0000","last_undegraded":"2023-07-14T19:03:30.209264+0000","last_fullsized":"2023-07-14T19:03:30.209264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:13:59.170240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.609207+0000","last_change":"2023-07-14T19:02:30.579680+0000","last_active":"2023-07-14T19:02:35.609207+0000","last_peered":"2023-07-14T19:02:35.609207+0000","last_clean":"2023-07-14T19:02:35.609207+0000","last_became_active":"2023-07-14T19:02:30.579379+0000","last_became_peered":"2023-07-14T19:02:30.579379+0000","last_unstale":"2023-07-14T19:02:35.609207+0000","last_undegraded":"2023-07-14T19:02:35.609207+0000","last_fullsized":"2023-07-14T19:02:35.609207+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:37:18.570317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.811962+0000","last_change":"2023-07-14T19:02:30.577722+0000","last_active":"2023-07-14T19:02:35.811962+0000","last_peered":"2023-07-14T19:02:35.811962+0000","last_clean":"2023-07-14T19:02:35.811962+0000","last_became_active":"2023-07-14T19:02:30.577247+0000","last_became_peered":"2023-07-14T19:02:30.577247+0000","last_unstale":"2023-07-14T19:02:35.811962+0000","last_undegraded":"2023-07-14T19:02:35.811962+0000","last_fullsized":"2023-07-14T19:02:35.811962+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:29.058219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218436+0000","last_change":"2023-07-14T19:02:30.571052+0000","last_active":"2023-07-14T19:03:31.218436+0000","last_peered":"2023-07-14T19:03:31.218436+0000","last_clean":"2023-07-14T19:03:31.218436+0000","last_became_active":"2023-07-14T19:02:30.570722+0000","last_became_peered":"2023-07-14T19:02:30.570722+0000","last_unstale":"2023-07-14T19:03:31.218436+0000","last_undegraded":"2023-07-14T19:03:31.218436+0000","last_fullsized":"2023-07-14T19:03:31.218436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:50:37.718571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.812013+0000","last_change":"2023-07-14T19:02:30.575685+0000","last_active":"2023-07-14T19:02:35.812013+0000","last_peered":"2023-07-14T19:02:35.812013+0000","last_clean":"2023-07-14T19:02:35.812013+0000","last_became_active":"2023-07-14T19:02:30.575379+0000","last_became_peered":"2023-07-14T19:02:30.575379+0000","last_unstale":"2023-07-14T19:02:35.812013+0000","last_undegraded":"2023-07-14T19:02:35.812013+0000","last_fullsized":"2023-07-14T19:02:35.812013+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:24:31.763621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281456+0000","last_change":"2023-07-14T19:02:30.582450+0000","last_active":"2023-07-14T19:03:31.281456+0000","last_peered":"2023-07-14T19:03:31.281456+0000","last_clean":"2023-07-14T19:03:31.281456+0000","last_became_active":"2023-07-14T19:02:30.582074+0000","last_became_peered":"2023-07-14T19:02:30.582074+0000","last_unstale":"2023-07-14T19:03:31.281456+0000","last_undegraded":"2023-07-14T19:03:31.281456+0000","last_fullsized":"2023-07-14T19:03:31.281456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:50.575652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2023-07-14T19:02:35.608673+0000","last_change":"2023-07-14T19:02:30.573042+0000","last_active":"2023-07-14T19:02:35.608673+0000","last_peered":"2023-07-14T19:02:35.608673+0000","last_clean":"2023-07-14T19:02:35.608673+0000","last_became_active":"2023-07-14T19:02:30.572498+0000","last_became_peered":"2023-07-14T19:02:30.572498+0000","last_unstale":"2023-07-14T19:02:35.608673+0000","last_undegraded":"2023-07-14T19:02:35.608673+0000","last_fullsized":"2023-07-14T19:02:35.608673+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:42:33.413533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2023-07-14T19:03:30.208888+0000","last_change":"2023-07-14T19:02:30.569441+0000","last_active":"2023-07-14T19:03:30.208888+0000","last_peered":"2023-07-14T19:03:30.208888+0000","last_clean":"2023-07-14T19:03:30.208888+0000","last_became_active":"2023-07-14T19:02:30.569180+0000","last_became_peered":"2023-07-14T19:02:30.569180+0000","last_unstale":"2023-07-14T19:03:30.208888+0000","last_undegraded":"2023-07-14T19:03:30.208888+0000","last_fullsized":"2023-07-14T19:03:30.208888+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:34:29.430277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61264,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":12,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":279,"num_read_kb":204,"num_write":153,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514520,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":467112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.436},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.53900000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.47399999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.66300000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.622}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.76100000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.64200000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.69099999999999995}]},{"osd":8,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.81599999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.747}]}]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1654784,"data_stored":1121258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.439},{"interface":"front","average":{"1min":0.504,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.36399999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.63100000000000001,"15min":0.621},"min":{"1min":0.48499999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57299999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.442,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45400000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79000000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59999999999999998,"15min":0.60099999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.47099999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.56399999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.61399999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69499999999999995,"15min":0.69299999999999995},"min":{"1min":0.51200000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89100000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.60099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77800000000000002,"15min":0.77900000000000003},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96699999999999997,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.627}]},{"osd":9,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77800000000000002,"15min":0.78200000000000003},"min":{"1min":0.59599999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.747,"15min":0.753},"min":{"1min":0.57199999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88,"5min":1.226,"15min":1.226},"last":0.58999999999999997}]},{"osd":11,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.66700000000000004}]}]},{"osd":9,"up_from":64,"seq":274877906976,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":481010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.42099999999999999,"5min":0.63,"15min":0.63},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79000000000000004,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.71299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47699999999999998,"15min":0.49299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66300000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.34200000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.41099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72499999999999998,"5min":1.008,"15min":1.008},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.746,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63}]},{"osd":5,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70499999999999996,"5min":1.026,"15min":1.026},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.70099999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.62},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.69999999999999996,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.81799999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.73499999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.73699999999999999,"15min":0.755},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81399999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84299999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.59499999999999997}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.746,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.70399999999999996,"15min":0.71499999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79400000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.745}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135908,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":457427,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88300000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.48199999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90900000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76300000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.64700000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.55100000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.878,"15min":0.878},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.76000000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.68400000000000005,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.88600000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.626,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.84899999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.94299999999999995,"5min":1.01,"15min":1.01},"last":0.65400000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.871,"15min":0.871},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0369999999999999,"5min":1.046,"15min":1.046},"last":0.82799999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.755,"15min":0.75},"min":{"1min":0.53200000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.92100000000000004},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.626}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"min":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.1140000000000001,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.081,"5min":1.081,"15min":1.081},"last":0.98599999999999999}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79800000000000004}]}]},{"osd":7,"up_from":50,"seq":214748364841,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28544,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716608,"statfs":{"total":95995035648,"available":95965806592,"internally_reserved":0,"allocated":1560576,"data_stored":1063197,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56999999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.40899999999999997},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.65700000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57399999999999995,"15min":0.54900000000000004},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.61499999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71699999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46200000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.47699999999999998,"15min":0.45900000000000002},"min":{"1min":0.38300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.48599999999999999},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69999999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.5}]},{"osd":5,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.627,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.87,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.752}]},{"osd":6,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.63},"min":{"1min":0.54100000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94899999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71599999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.79000000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74099999999999999,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96999999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.997,"5min":0.76300000000000001,"15min":0.72399999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":6.8970000000000002,"5min":6.8970000000000002,"15min":6.8970000000000002},"last":0.69399999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.72299999999999998},"min":{"1min":0.60599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95299999999999996,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.84499999999999997},{"interface":"front","average":{"1min":0.745,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.64700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.81399999999999995}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.86399999999999999},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.83199999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":453014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39900000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.54700000000000004,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41999999999999998,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56200000000000006,"5min":1.591,"15min":1.591},"last":0.27700000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.55700000000000005},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83699999999999997,"5min":1.159,"15min":1.159},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":1.49,"15min":1.49},"last":0.53000000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73799999999999999,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61099999999999999,"15min":0.59299999999999997},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.77900000000000003,"5min":1.419,"15min":1.419},"last":0.41799999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.34300000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.48199999999999998,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":2.077,"15min":2.077},"last":0.35999999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85299999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64600000000000002,"15min":0.624},"min":{"1min":0.45500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80900000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.72999999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65100000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76200000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.51200000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:02:31 2023","interfaces":[{"interface":"back","average":{"1min":0.69299999999999995,"5min":0.68600000000000005,"15min":0.68500000000000005},"min":{"1min":0.46600000000000003,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.2529999999999999,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.76200000000000001,"15min":0.73499999999999999},"min":{"1min":0.52600000000000002,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":3.6709999999999998,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.69599999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73599999999999999},"min":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.77400000000000002,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78900000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.56299999999999994}]},{"osd":10,"last update":"Fri Jul 14 19:02:31 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.2829999999999999,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.628},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.234,"5min":1.234,"15min":1.234},"last":0.78900000000000003}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":528970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52400000000000002,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.3370000000000002},"last":0.378},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.379,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.55200000000000005,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.52500000000000002,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88300000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81999999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.49399999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.44600000000000001,"15min":0.39100000000000001},"min":{"1min":0.36599999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44600000000000001,"15min":0.40200000000000002},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.39700000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61799999999999999,"15min":0.59199999999999997},"min":{"1min":0.53800000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.433},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.65000000000000002,"15min":0.60199999999999998},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.3209999999999997,"5min":4.3209999999999997,"15min":4.3209999999999997},"last":0.53800000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.54900000000000004,"15min":0.51700000000000002},"min":{"1min":0.439,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.628,"15min":0.58199999999999996},"min":{"1min":0.52200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.52200000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72899999999999998,"15min":0.71599999999999997},"min":{"1min":0.498,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.99299999999999999,"5min":0.996,"15min":0.996},"last":0.57099999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.79000000000000004,"15min":0.77300000000000002},"min":{"1min":0.54800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.60099999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.66000000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.77000000000000002,"15min":0.76000000000000001},"min":{"1min":0.625,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.625},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.73199999999999998,"15min":0.72499999999999998},"min":{"1min":0.60699999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.90300000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.60699999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.63600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":463639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.159,"15min":0.159},"max":{"1min":0.72499999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.48999999999999999},"min":{"1min":0.40300000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.75800000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.37,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.746,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.41299999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.435,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.29499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.32700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.372,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68700000000000006,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.627},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.50900000000000001},"min":{"1min":0.43099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.73899999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.46000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91700000000000004,"5min":1.042,"15min":1.042},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.496,"15min":0.495},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72099999999999997,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.67000000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60199999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70199999999999996,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.024,"15min":1.024},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.57999999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.78200000000000003,"5min":1.204,"15min":1.204},"last":0.503},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72499999999999998,"15min":0.746},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82199999999999995,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63900000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.877,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.79700000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.71499999999999997}]},{"osd":11,"last update":"Fri Jul 14 19:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66900000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411390,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":511586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78700000000000003,"5min":1.111,"15min":1.111},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.435}]},{"osd":1,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.628,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.71599999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77000000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55200000000000005,"15min":0.53900000000000003},"min":{"1min":0.45500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78100000000000003,"5min":1.046,"15min":1.046},"last":0.52900000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:01:26 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.54300000000000004,"15min":0.54000000000000004},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.56599999999999995},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"last":0.59699999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.44500000000000001,"15min":0.41799999999999998},"min":{"1min":0.33600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.63400000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.59599999999999997,"15min":0.56599999999999995},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":1.4359999999999999},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60299999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89300000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.69599999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60199999999999998,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.748},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61599999999999999,"15min":0.60699999999999998},"min":{"1min":0.47499999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.79600000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.443,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":1.204},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.59599999999999997,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.68600000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92600000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72799999999999998,"15min":0.71999999999999997},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.91300000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.94799999999999995},{"interface":"front","average":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.81399999999999995}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.85899999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":453208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.188,"15min":0.188},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.253},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35299999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.215,"15min":0.215},"max":{"1min":0.52800000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.26900000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71799999999999997,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.497}]},{"osd":2,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.39600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.745,"5min":4.6509999999999998,"15min":4.6509999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60799999999999998,"15min":0.66400000000000003},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.48399999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45700000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.498,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.59099999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.82799999999999996,"5min":1.119,"15min":1.119},"last":0.33500000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:02:27 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.42799999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.76500000000000001,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.64000000000000001,"5min":0.60399999999999998,"15min":0.59799999999999998},"min":{"1min":0.48999999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.83299999999999996,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.60899999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66200000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76000000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.372}]},{"osd":10,"last update":"Fri Jul 14 19:02:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":0.90000000000000002,"5min":0.90000000000000002,"15min":0.90000000000000002},"last":0.626}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57899999999999996}]}]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":476809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.503,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.38100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38400000000000001,"15min":0.34799999999999998},"min":{"1min":0.33600000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.59199999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.44},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40100000000000002,"15min":0.36499999999999999},"min":{"1min":0.34399999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.71799999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.376}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.89500000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.58499999999999996,"15min":0.54300000000000004},"min":{"1min":0.45500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.105,"15min":1.105},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55300000000000005,"15min":0.51200000000000001},"min":{"1min":0.432,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72299999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54900000000000004,"15min":0.50600000000000001},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48199999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57599999999999996,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.69599999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.60399999999999998},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":1.038,"15min":1.038},"last":0.59099999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.442,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.873,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68600000000000005,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.75600000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.58399999999999996,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77600000000000002}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.81000000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1067951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.495},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.41399999999999998,"15min":0.38900000000000001},"min":{"1min":0.309,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70599999999999996,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60199999999999998,"15min":0.58799999999999997},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84099999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64500000000000002,"15min":0.63100000000000001},"min":{"1min":0.40200000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.60699999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63900000000000001,"15min":0.622},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.65100000000000002,"15min":0.64000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.64700000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.71699999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.753},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.66300000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.755},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.73399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":58662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:03:35.981 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:03:36.531 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:36.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:36 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:03:36.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:36 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:03:36.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:36 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:36.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:36 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:03:36.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:36 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:03:36.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:36 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:36.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:36 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:03:36.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:36 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:03:36.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:36 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:03:37.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:37 smithi078 ceph-mon[127425]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:37.511 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:37 smithi078 ceph-mon[127425]: pgmap v290: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 18 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:37.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:37 smithi102 ceph-mon[129280]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:37.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:37 smithi102 ceph-mon[129280]: pgmap v290: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 18 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:37.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:37 smithi088 ceph-mon[128973]: from='client.15102 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:37.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:37 smithi088 ceph-mon[128973]: pgmap v290: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s; 18 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2106876873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 msgr2=0x7f61b4104ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2106876873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 0x7f61b4104ff0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009990 tx=0x7f61a4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2106876873 shutdown_connections 2023-07-14T19:03:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2106876873 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61b40feb30 0x7f61b4103920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2106876873 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 0x7f61b4104ff0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2106876873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b40fe180 0x7f61b40fe5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2106876873 >> 172.21.15.78:0/2106876873 conn(0x7f61b4075050 msgr2=0x7f61b4075450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2106876873 shutdown_connections 2023-07-14T19:03:37.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.710+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2106876873 wait complete. 2023-07-14T19:03:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.711+0000 7f61ba7dc700 1 Processor -- start 2023-07-14T19:03:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.711+0000 7f61ba7dc700 1 -- start start 2023-07-14T19:03:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.711+0000 7f61ba7dc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b40fe180 0x7f61b419c240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.711+0000 7f61ba7dc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61b40feb30 0x7f61b419c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:37.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.711+0000 7f61ba7dc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 0x7f61b41a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.711+0000 7f61ba7dc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41a1200 con 0x7f61b4104c20 2023-07-14T19:03:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.711+0000 7f61ba7dc700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41a1370 con 0x7f61b40feb30 2023-07-14T19:03:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.711+0000 7f61ba7dc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b41a1530 con 0x7f61b40fe180 2023-07-14T19:03:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 0x7f61b41a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 0x7f61b41a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35474/0 (socket says 172.21.15.78:35474) 2023-07-14T19:03:37.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 -- 172.21.15.78:0/2537494888 learned_addr learned my addr 172.21.15.78:0/2537494888 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b37fe700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61b40feb30 0x7f61b419c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b3fff700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b40fe180 0x7f61b419c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 -- 172.21.15.78:0/2537494888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b40fe180 msgr2=0x7f61b419c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:37.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b40fe180 0x7f61b419c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 -- 172.21.15.78:0/2537494888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61b40feb30 msgr2=0x7f61b419c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61b40feb30 0x7f61b419c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 -- 172.21.15.78:0/2537494888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a40095f0 con 0x7f61b4104c20 2023-07-14T19:03:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b8d79700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 0x7f61b41a0bf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f61ac007ee0 tx=0x7f61ac00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b17fa700 1 -- 172.21.15.78:0/2537494888 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac017070 con 0x7f61b4104c20 2023-07-14T19:03:37.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b41a1810 con 0x7f61b4104c20 2023-07-14T19:03:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b17fa700 1 -- 172.21.15.78:0/2537494888 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f61ac00f040 con 0x7f61b4104c20 2023-07-14T19:03:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61b17fa700 1 -- 172.21.15.78:0/2537494888 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61ac005710 con 0x7f61b4104c20 2023-07-14T19:03:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.712+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b41a9070 con 0x7f61b4104c20 2023-07-14T19:03:37.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.713+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b404e340 con 0x7f61b4104c20 2023-07-14T19:03:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.714+0000 7f61b17fa700 1 -- 172.21.15.78:0/2537494888 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f61ac005870 con 0x7f61b4104c20 2023-07-14T19:03:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.715+0000 7f61b17fa700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f619c0a11e0 0x7f619c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.715+0000 7f61b3fff700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f619c0a11e0 0x7f619c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.715+0000 7f61b17fa700 1 -- 172.21.15.78:0/2537494888 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f61ac0ccbf0 con 0x7f61b4104c20 2023-07-14T19:03:37.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.716+0000 7f61b3fff700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f619c0a11e0 0x7f619c0a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f61a400b5c0 tx=0x7f61a4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:37.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.719+0000 7f61b17fa700 1 -- 172.21.15.78:0/2537494888 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f61ac095ed0 con 0x7f61b4104c20 2023-07-14T19:03:37.905 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.901+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f61b41030f0 con 0x7f619c0a11e0 2023-07-14T19:03:37.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.913+0000 7f61b17fa700 1 -- 172.21.15.78:0/2537494888 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+426221 (secure 0 0 0) 0x7f61b41030f0 con 0x7f619c0a11e0 2023-07-14T19:03:37.921 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f619c0a11e0 msgr2=0x7f619c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f619c0a11e0 0x7f619c0a3690 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f61a400b5c0 tx=0x7f61a4005fb0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 msgr2=0x7f61b41a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:37.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 0x7f61b41a0bf0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f61ac007ee0 tx=0x7f61ac00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 shutdown_connections 2023-07-14T19:03:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f619c0a11e0 0x7f619c0a3690 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f61b4104c20 0x7f61b41a0bf0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61b40fe180 0x7f61b419c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 --2- 172.21.15.78:0/2537494888 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f61b40feb30 0x7f61b419c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 >> 172.21.15.78:0/2537494888 conn(0x7f61b4075050 msgr2=0x7f61b41019d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:37.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 shutdown_connections 2023-07-14T19:03:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:37.917+0000 7f61ba7dc700 1 -- 172.21.15.78:0/2537494888 wait complete. 2023-07-14T19:03:37.925 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-14T19:03:38.389 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":290,"stamp":"2023-07-14T19:03:36.126231+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651878,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":410,"num_read_kb":318,"num_write":336,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1180882,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":425,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337680,"kb_used_data":13412,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604144,"statfs":{"total":1151940427776,"available":1151594643456,"internally_reserved":0,"allocated":13733888,"data_stored":7640617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":49,"num_read_kb":29,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":146,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.002075"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210016+0000","last_change":"2023-07-14T19:02:30.582427+0000","last_active":"2023-07-14T19:03:31.210016+0000","last_peered":"2023-07-14T19:03:31.210016+0000","last_clean":"2023-07-14T19:03:31.210016+0000","last_became_active":"2023-07-14T19:02:30.582149+0000","last_became_peered":"2023-07-14T19:02:30.582149+0000","last_unstale":"2023-07-14T19:03:31.210016+0000","last_undegraded":"2023-07-14T19:03:31.210016+0000","last_fullsized":"2023-07-14T19:03:31.210016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:09.726556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208324+0000","last_change":"2023-07-14T19:02:30.567053+0000","last_active":"2023-07-14T19:03:31.208324+0000","last_peered":"2023-07-14T19:03:31.208324+0000","last_clean":"2023-07-14T19:03:31.208324+0000","last_became_active":"2023-07-14T19:02:30.566736+0000","last_became_peered":"2023-07-14T19:02:30.566736+0000","last_unstale":"2023-07-14T19:03:31.208324+0000","last_undegraded":"2023-07-14T19:03:31.208324+0000","last_fullsized":"2023-07-14T19:03:31.208324+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:44:23.099904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208244+0000","last_change":"2023-07-14T19:02:30.577632+0000","last_active":"2023-07-14T19:03:31.208244+0000","last_peered":"2023-07-14T19:03:31.208244+0000","last_clean":"2023-07-14T19:03:31.208244+0000","last_became_active":"2023-07-14T19:02:30.577300+0000","last_became_peered":"2023-07-14T19:02:30.577300+0000","last_unstale":"2023-07-14T19:03:31.208244+0000","last_undegraded":"2023-07-14T19:03:31.208244+0000","last_fullsized":"2023-07-14T19:03:31.208244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:15:18.165242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213866+0000","last_change":"2023-07-14T19:02:30.581405+0000","last_active":"2023-07-14T19:03:31.213866+0000","last_peered":"2023-07-14T19:03:31.213866+0000","last_clean":"2023-07-14T19:03:31.213866+0000","last_became_active":"2023-07-14T19:02:30.581113+0000","last_became_peered":"2023-07-14T19:02:30.581113+0000","last_unstale":"2023-07-14T19:03:31.213866+0000","last_undegraded":"2023-07-14T19:03:31.213866+0000","last_fullsized":"2023-07-14T19:03:31.213866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:42:53.074489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212421+0000","last_change":"2023-07-14T19:02:30.574285+0000","last_active":"2023-07-14T19:03:31.212421+0000","last_peered":"2023-07-14T19:03:31.212421+0000","last_clean":"2023-07-14T19:03:31.212421+0000","last_became_active":"2023-07-14T19:02:30.573967+0000","last_became_peered":"2023-07-14T19:02:30.573967+0000","last_unstale":"2023-07-14T19:03:31.212421+0000","last_undegraded":"2023-07-14T19:03:31.212421+0000","last_fullsized":"2023-07-14T19:03:31.212421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:39:39.079558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215131+0000","last_change":"2023-07-14T19:02:30.574135+0000","last_active":"2023-07-14T19:03:31.215131+0000","last_peered":"2023-07-14T19:03:31.215131+0000","last_clean":"2023-07-14T19:03:31.215131+0000","last_became_active":"2023-07-14T19:02:30.573821+0000","last_became_peered":"2023-07-14T19:02:30.573821+0000","last_unstale":"2023-07-14T19:03:31.215131+0000","last_undegraded":"2023-07-14T19:03:31.215131+0000","last_fullsized":"2023-07-14T19:03:31.215131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:56.064229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212782+0000","last_change":"2023-07-14T19:02:30.578051+0000","last_active":"2023-07-14T19:03:31.212782+0000","last_peered":"2023-07-14T19:03:31.212782+0000","last_clean":"2023-07-14T19:03:31.212782+0000","last_became_active":"2023-07-14T19:02:30.577700+0000","last_became_peered":"2023-07-14T19:02:30.577700+0000","last_unstale":"2023-07-14T19:03:31.212782+0000","last_undegraded":"2023-07-14T19:03:31.212782+0000","last_fullsized":"2023-07-14T19:03:31.212782+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:40:01.289258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213306+0000","last_change":"2023-07-14T19:02:30.567480+0000","last_active":"2023-07-14T19:03:31.213306+0000","last_peered":"2023-07-14T19:03:31.213306+0000","last_clean":"2023-07-14T19:03:31.213306+0000","last_became_active":"2023-07-14T19:02:30.567147+0000","last_became_peered":"2023-07-14T19:02:30.567147+0000","last_unstale":"2023-07-14T19:03:31.213306+0000","last_undegraded":"2023-07-14T19:03:31.213306+0000","last_fullsized":"2023-07-14T19:03:31.213306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:02:57.085021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281462+0000","last_change":"2023-07-14T19:02:30.581936+0000","last_active":"2023-07-14T19:03:31.281462+0000","last_peered":"2023-07-14T19:03:31.281462+0000","last_clean":"2023-07-14T19:03:31.281462+0000","last_became_active":"2023-07-14T19:02:30.579619+0000","last_became_peered":"2023-07-14T19:02:30.579619+0000","last_unstale":"2023-07-14T19:03:31.281462+0000","last_undegraded":"2023-07-14T19:03:31.281462+0000","last_fullsized":"2023-07-14T19:03:31.281462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:23.438941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218216+0000","last_change":"2023-07-14T19:03:31.218216+0000","last_active":"2023-07-14T19:03:31.218216+0000","last_peered":"2023-07-14T19:03:31.218216+0000","last_clean":"2023-07-14T19:03:31.218216+0000","last_became_active":"2023-07-14T19:03:31.217871+0000","last_became_peered":"2023-07-14T19:03:31.217871+0000","last_unstale":"2023-07-14T19:03:31.218216+0000","last_undegraded":"2023-07-14T19:03:31.218216+0000","last_fullsized":"2023-07-14T19:03:31.218216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:56:32.678369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212386+0000","last_change":"2023-07-14T19:02:30.583662+0000","last_active":"2023-07-14T19:03:31.212386+0000","last_peered":"2023-07-14T19:03:31.212386+0000","last_clean":"2023-07-14T19:03:31.212386+0000","last_became_active":"2023-07-14T19:02:30.583352+0000","last_became_peered":"2023-07-14T19:02:30.583352+0000","last_unstale":"2023-07-14T19:03:31.212386+0000","last_undegraded":"2023-07-14T19:03:31.212386+0000","last_fullsized":"2023-07-14T19:03:31.212386+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:01:13.319863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213712+0000","last_change":"2023-07-14T19:02:30.567685+0000","last_active":"2023-07-14T19:03:31.213712+0000","last_peered":"2023-07-14T19:03:31.213712+0000","last_clean":"2023-07-14T19:03:31.213712+0000","last_became_active":"2023-07-14T19:02:30.566674+0000","last_became_peered":"2023-07-14T19:02:30.566674+0000","last_unstale":"2023-07-14T19:03:31.213712+0000","last_undegraded":"2023-07-14T19:03:31.213712+0000","last_fullsized":"2023-07-14T19:03:31.213712+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:41:27.460945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210143+0000","last_change":"2023-07-14T19:02:30.558012+0000","last_active":"2023-07-14T19:03:31.210143+0000","last_peered":"2023-07-14T19:03:31.210143+0000","last_clean":"2023-07-14T19:03:31.210143+0000","last_became_active":"2023-07-14T19:02:30.557675+0000","last_became_peered":"2023-07-14T19:02:30.557675+0000","last_unstale":"2023-07-14T19:03:31.210143+0000","last_undegraded":"2023-07-14T19:03:31.210143+0000","last_fullsized":"2023-07-14T19:03:31.210143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:55:41.257017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207948+0000","last_change":"2023-07-14T19:02:30.570341+0000","last_active":"2023-07-14T19:03:31.207948+0000","last_peered":"2023-07-14T19:03:31.207948+0000","last_clean":"2023-07-14T19:03:31.207948+0000","last_became_active":"2023-07-14T19:02:30.570091+0000","last_became_peered":"2023-07-14T19:02:30.570091+0000","last_unstale":"2023-07-14T19:03:31.207948+0000","last_undegraded":"2023-07-14T19:03:31.207948+0000","last_fullsized":"2023-07-14T19:03:31.207948+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:52:13.273335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207886+0000","last_change":"2023-07-14T19:02:30.573746+0000","last_active":"2023-07-14T19:03:31.207886+0000","last_peered":"2023-07-14T19:03:31.207886+0000","last_clean":"2023-07-14T19:03:31.207886+0000","last_became_active":"2023-07-14T19:02:30.573307+0000","last_became_peered":"2023-07-14T19:02:30.573307+0000","last_unstale":"2023-07-14T19:03:31.207886+0000","last_undegraded":"2023-07-14T19:03:31.207886+0000","last_fullsized":"2023-07-14T19:03:31.207886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:49.299962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.280974+0000","last_change":"2023-07-14T19:02:30.578352+0000","last_active":"2023-07-14T19:03:31.280974+0000","last_peered":"2023-07-14T19:03:31.280974+0000","last_clean":"2023-07-14T19:03:31.280974+0000","last_became_active":"2023-07-14T19:02:30.577631+0000","last_became_peered":"2023-07-14T19:02:30.577631+0000","last_unstale":"2023-07-14T19:03:31.280974+0000","last_undegraded":"2023-07-14T19:03:31.280974+0000","last_fullsized":"2023-07-14T19:03:31.280974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:07:01.573189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209912+0000","last_change":"2023-07-14T19:02:30.577813+0000","last_active":"2023-07-14T19:03:31.209912+0000","last_peered":"2023-07-14T19:03:31.209912+0000","last_clean":"2023-07-14T19:03:31.209912+0000","last_became_active":"2023-07-14T19:02:30.577482+0000","last_became_peered":"2023-07-14T19:02:30.577482+0000","last_unstale":"2023-07-14T19:03:31.209912+0000","last_undegraded":"2023-07-14T19:03:31.209912+0000","last_fullsized":"2023-07-14T19:03:31.209912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:11:33.046274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210222+0000","last_change":"2023-07-14T19:02:30.583353+0000","last_active":"2023-07-14T19:03:31.210222+0000","last_peered":"2023-07-14T19:03:31.210222+0000","last_clean":"2023-07-14T19:03:31.210222+0000","last_became_active":"2023-07-14T19:02:30.583037+0000","last_became_peered":"2023-07-14T19:02:30.583037+0000","last_unstale":"2023-07-14T19:03:31.210222+0000","last_undegraded":"2023-07-14T19:03:31.210222+0000","last_fullsized":"2023-07-14T19:03:31.210222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:43:40.021757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214491+0000","last_change":"2023-07-14T19:02:30.569788+0000","last_active":"2023-07-14T19:03:31.214491+0000","last_peered":"2023-07-14T19:03:31.214491+0000","last_clean":"2023-07-14T19:03:31.214491+0000","last_became_active":"2023-07-14T19:02:30.569461+0000","last_became_peered":"2023-07-14T19:02:30.569461+0000","last_unstale":"2023-07-14T19:03:31.214491+0000","last_undegraded":"2023-07-14T19:03:31.214491+0000","last_fullsized":"2023-07-14T19:03:31.214491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:49.917455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214902+0000","last_change":"2023-07-14T19:02:34.585220+0000","last_active":"2023-07-14T19:03:31.214902+0000","last_peered":"2023-07-14T19:03:31.214902+0000","last_clean":"2023-07-14T19:03:31.214902+0000","last_became_active":"2023-07-14T19:02:34.584771+0000","last_became_peered":"2023-07-14T19:02:34.584771+0000","last_unstale":"2023-07-14T19:03:31.214902+0000","last_undegraded":"2023-07-14T19:03:31.214902+0000","last_fullsized":"2023-07-14T19:03:31.214902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:38.451682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210698+0000","last_change":"2023-07-14T19:02:29.938571+0000","last_active":"2023-07-14T19:03:31.210698+0000","last_peered":"2023-07-14T19:03:31.210698+0000","last_clean":"2023-07-14T19:03:31.210698+0000","last_became_active":"2023-07-14T19:02:29.938270+0000","last_became_peered":"2023-07-14T19:02:29.938270+0000","last_unstale":"2023-07-14T19:03:31.210698+0000","last_undegraded":"2023-07-14T19:03:31.210698+0000","last_fullsized":"2023-07-14T19:03:31.210698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:02.148507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214143+0000","last_change":"2023-07-14T19:02:30.575894+0000","last_active":"2023-07-14T19:03:31.214143+0000","last_peered":"2023-07-14T19:03:31.214143+0000","last_clean":"2023-07-14T19:03:31.214143+0000","last_became_active":"2023-07-14T19:02:30.575607+0000","last_became_peered":"2023-07-14T19:02:30.575607+0000","last_unstale":"2023-07-14T19:03:31.214143+0000","last_undegraded":"2023-07-14T19:03:31.214143+0000","last_fullsized":"2023-07-14T19:03:31.214143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:37.751234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218192+0000","last_change":"2023-07-14T19:03:31.218192+0000","last_active":"2023-07-14T19:03:31.218192+0000","last_peered":"2023-07-14T19:03:31.218192+0000","last_clean":"2023-07-14T19:03:31.218192+0000","last_became_active":"2023-07-14T19:03:31.217708+0000","last_became_peered":"2023-07-14T19:03:31.217708+0000","last_unstale":"2023-07-14T19:03:31.218192+0000","last_undegraded":"2023-07-14T19:03:31.218192+0000","last_fullsized":"2023-07-14T19:03:31.218192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:40.671215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214160+0000","last_change":"2023-07-14T19:02:29.944136+0000","last_active":"2023-07-14T19:03:31.214160+0000","last_peered":"2023-07-14T19:03:31.214160+0000","last_clean":"2023-07-14T19:03:31.214160+0000","last_became_active":"2023-07-14T19:02:29.943751+0000","last_became_peered":"2023-07-14T19:02:29.943751+0000","last_unstale":"2023-07-14T19:03:31.214160+0000","last_undegraded":"2023-07-14T19:03:31.214160+0000","last_fullsized":"2023-07-14T19:03:31.214160+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:43:53.346383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208280+0000","last_change":"2023-07-14T19:02:30.583156+0000","last_active":"2023-07-14T19:03:31.208280+0000","last_peered":"2023-07-14T19:03:31.208280+0000","last_clean":"2023-07-14T19:03:31.208280+0000","last_became_active":"2023-07-14T19:02:30.582794+0000","last_became_peered":"2023-07-14T19:02:30.582794+0000","last_unstale":"2023-07-14T19:03:31.208280+0000","last_undegraded":"2023-07-14T19:03:31.208280+0000","last_fullsized":"2023-07-14T19:03:31.208280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:47:47.558145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214058+0000","last_change":"2023-07-14T19:02:34.590788+0000","last_active":"2023-07-14T19:03:31.214058+0000","last_peered":"2023-07-14T19:03:31.214058+0000","last_clean":"2023-07-14T19:03:31.214058+0000","last_became_active":"2023-07-14T19:02:34.590385+0000","last_became_peered":"2023-07-14T19:02:34.590385+0000","last_unstale":"2023-07-14T19:03:31.214058+0000","last_undegraded":"2023-07-14T19:03:31.214058+0000","last_fullsized":"2023-07-14T19:03:31.214058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:35.914846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.222715+0000","last_change":"2023-07-14T19:03:31.222715+0000","last_active":"2023-07-14T19:03:31.222715+0000","last_peered":"2023-07-14T19:03:31.222715+0000","last_clean":"2023-07-14T19:03:31.222715+0000","last_became_active":"2023-07-14T19:03:31.216343+0000","last_became_peered":"2023-07-14T19:03:31.216343+0000","last_unstale":"2023-07-14T19:03:31.222715+0000","last_undegraded":"2023-07-14T19:03:31.222715+0000","last_fullsized":"2023-07-14T19:03:31.222715+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:10.973001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281375+0000","last_change":"2023-07-14T19:02:30.582746+0000","last_active":"2023-07-14T19:03:31.281375+0000","last_peered":"2023-07-14T19:03:31.281375+0000","last_clean":"2023-07-14T19:03:31.281375+0000","last_became_active":"2023-07-14T19:02:30.582426+0000","last_became_peered":"2023-07-14T19:02:30.582426+0000","last_unstale":"2023-07-14T19:03:31.281375+0000","last_undegraded":"2023-07-14T19:03:31.281375+0000","last_fullsized":"2023-07-14T19:03:31.281375+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:16:53.120645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212948+0000","last_change":"2023-07-14T19:02:34.589966+0000","last_active":"2023-07-14T19:03:31.212948+0000","last_peered":"2023-07-14T19:03:31.212948+0000","last_clean":"2023-07-14T19:03:31.212948+0000","last_became_active":"2023-07-14T19:02:34.589544+0000","last_became_peered":"2023-07-14T19:02:34.589544+0000","last_unstale":"2023-07-14T19:03:31.212948+0000","last_undegraded":"2023-07-14T19:03:31.212948+0000","last_fullsized":"2023-07-14T19:03:31.212948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:08.813030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210414+0000","last_change":"2023-07-14T19:02:29.935507+0000","last_active":"2023-07-14T19:03:31.210414+0000","last_peered":"2023-07-14T19:03:31.210414+0000","last_clean":"2023-07-14T19:03:31.210414+0000","last_became_active":"2023-07-14T19:02:29.935160+0000","last_became_peered":"2023-07-14T19:02:29.935160+0000","last_unstale":"2023-07-14T19:03:31.210414+0000","last_undegraded":"2023-07-14T19:03:31.210414+0000","last_fullsized":"2023-07-14T19:03:31.210414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:02.794267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219342+0000","last_change":"2023-07-14T19:02:30.578077+0000","last_active":"2023-07-14T19:03:31.219342+0000","last_peered":"2023-07-14T19:03:31.219342+0000","last_clean":"2023-07-14T19:03:31.219342+0000","last_became_active":"2023-07-14T19:02:30.577764+0000","last_became_peered":"2023-07-14T19:02:30.577764+0000","last_unstale":"2023-07-14T19:03:31.219342+0000","last_undegraded":"2023-07-14T19:03:31.219342+0000","last_fullsized":"2023-07-14T19:03:31.219342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:40:14.188240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214161+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:03:31.214161+0000","last_peered":"2023-07-14T19:03:31.214161+0000","last_clean":"2023-07-14T19:03:31.214161+0000","last_became_active":"2023-07-14T19:02:34.583408+0000","last_became_peered":"2023-07-14T19:02:34.583408+0000","last_unstale":"2023-07-14T19:03:31.214161+0000","last_undegraded":"2023-07-14T19:03:31.214161+0000","last_fullsized":"2023-07-14T19:03:31.214161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:10.318901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.220191+0000","last_change":"2023-07-14T19:02:29.943899+0000","last_active":"2023-07-14T19:03:31.220191+0000","last_peered":"2023-07-14T19:03:31.220191+0000","last_clean":"2023-07-14T19:03:31.220191+0000","last_became_active":"2023-07-14T19:02:29.943585+0000","last_became_peered":"2023-07-14T19:02:29.943585+0000","last_unstale":"2023-07-14T19:03:31.220191+0000","last_undegraded":"2023-07-14T19:03:31.220191+0000","last_fullsized":"2023-07-14T19:03:31.220191+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:29.940165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210875+0000","last_change":"2023-07-14T19:02:30.567147+0000","last_active":"2023-07-14T19:03:31.210875+0000","last_peered":"2023-07-14T19:03:31.210875+0000","last_clean":"2023-07-14T19:03:31.210875+0000","last_became_active":"2023-07-14T19:02:30.566792+0000","last_became_peered":"2023-07-14T19:02:30.566792+0000","last_unstale":"2023-07-14T19:03:31.210875+0000","last_undegraded":"2023-07-14T19:03:31.210875+0000","last_fullsized":"2023-07-14T19:03:31.210875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:00:27.150739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214266+0000","last_change":"2023-07-14T19:02:34.579561+0000","last_active":"2023-07-14T19:03:31.214266+0000","last_peered":"2023-07-14T19:03:31.214266+0000","last_clean":"2023-07-14T19:03:31.214266+0000","last_became_active":"2023-07-14T19:02:34.579099+0000","last_became_peered":"2023-07-14T19:02:34.579099+0000","last_unstale":"2023-07-14T19:03:31.214266+0000","last_undegraded":"2023-07-14T19:03:31.214266+0000","last_fullsized":"2023-07-14T19:03:31.214266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:25:32.422783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.284881+0000","last_change":"2023-07-14T19:03:31.284881+0000","last_active":"2023-07-14T19:03:31.284881+0000","last_peered":"2023-07-14T19:03:31.284881+0000","last_clean":"2023-07-14T19:03:31.284881+0000","last_became_active":"2023-07-14T19:03:31.282061+0000","last_became_peered":"2023-07-14T19:03:31.282061+0000","last_unstale":"2023-07-14T19:03:31.284881+0000","last_undegraded":"2023-07-14T19:03:31.284881+0000","last_fullsized":"2023-07-14T19:03:31.284881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:32.928291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.220085+0000","last_change":"2023-07-14T19:02:30.575925+0000","last_active":"2023-07-14T19:03:31.220085+0000","last_peered":"2023-07-14T19:03:31.220085+0000","last_clean":"2023-07-14T19:03:31.220085+0000","last_became_active":"2023-07-14T19:02:30.575583+0000","last_became_peered":"2023-07-14T19:02:30.575583+0000","last_unstale":"2023-07-14T19:03:31.220085+0000","last_undegraded":"2023-07-14T19:03:31.220085+0000","last_fullsized":"2023-07-14T19:03:31.220085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:11:02.245125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209772+0000","last_change":"2023-07-14T19:02:34.587094+0000","last_active":"2023-07-14T19:03:31.209772+0000","last_peered":"2023-07-14T19:03:31.209772+0000","last_clean":"2023-07-14T19:03:31.209772+0000","last_became_active":"2023-07-14T19:02:34.586650+0000","last_became_peered":"2023-07-14T19:02:34.586650+0000","last_unstale":"2023-07-14T19:03:31.209772+0000","last_undegraded":"2023-07-14T19:03:31.209772+0000","last_fullsized":"2023-07-14T19:03:31.209772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:57.546314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209853+0000","last_change":"2023-07-14T19:02:29.934651+0000","last_active":"2023-07-14T19:03:31.209853+0000","last_peered":"2023-07-14T19:03:31.209853+0000","last_clean":"2023-07-14T19:03:31.209853+0000","last_became_active":"2023-07-14T19:02:29.934347+0000","last_became_peered":"2023-07-14T19:02:29.934347+0000","last_unstale":"2023-07-14T19:03:31.209853+0000","last_undegraded":"2023-07-14T19:03:31.209853+0000","last_fullsized":"2023-07-14T19:03:31.209853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:53:44.964924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213264+0000","last_change":"2023-07-14T19:02:30.571680+0000","last_active":"2023-07-14T19:03:31.213264+0000","last_peered":"2023-07-14T19:03:31.213264+0000","last_clean":"2023-07-14T19:03:31.213264+0000","last_became_active":"2023-07-14T19:02:30.571367+0000","last_became_peered":"2023-07-14T19:02:30.571367+0000","last_unstale":"2023-07-14T19:03:31.213264+0000","last_undegraded":"2023-07-14T19:03:31.213264+0000","last_fullsized":"2023-07-14T19:03:31.213264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:53.039270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208272+0000","last_change":"2023-07-14T19:02:34.577642+0000","last_active":"2023-07-14T19:03:31.208272+0000","last_peered":"2023-07-14T19:03:31.208272+0000","last_clean":"2023-07-14T19:03:31.208272+0000","last_became_active":"2023-07-14T19:02:34.577232+0000","last_became_peered":"2023-07-14T19:02:34.577232+0000","last_unstale":"2023-07-14T19:03:31.208272+0000","last_undegraded":"2023-07-14T19:03:31.208272+0000","last_fullsized":"2023-07-14T19:03:31.208272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:19.952531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218302+0000","last_change":"2023-07-14T19:02:29.940688+0000","last_active":"2023-07-14T19:03:31.218302+0000","last_peered":"2023-07-14T19:03:31.218302+0000","last_clean":"2023-07-14T19:03:31.218302+0000","last_became_active":"2023-07-14T19:02:29.940444+0000","last_became_peered":"2023-07-14T19:02:29.940444+0000","last_unstale":"2023-07-14T19:03:31.218302+0000","last_undegraded":"2023-07-14T19:03:31.218302+0000","last_fullsized":"2023-07-14T19:03:31.218302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:37.769743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210635+0000","last_change":"2023-07-14T19:02:30.568127+0000","last_active":"2023-07-14T19:03:31.210635+0000","last_peered":"2023-07-14T19:03:31.210635+0000","last_clean":"2023-07-14T19:03:31.210635+0000","last_became_active":"2023-07-14T19:02:30.567811+0000","last_became_peered":"2023-07-14T19:02:30.567811+0000","last_unstale":"2023-07-14T19:03:31.210635+0000","last_undegraded":"2023-07-14T19:03:31.210635+0000","last_fullsized":"2023-07-14T19:03:31.210635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:56.545251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213502+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:03:31.213502+0000","last_peered":"2023-07-14T19:03:31.213502+0000","last_clean":"2023-07-14T19:03:31.213502+0000","last_became_active":"2023-07-14T19:02:34.583480+0000","last_became_peered":"2023-07-14T19:02:34.583480+0000","last_unstale":"2023-07-14T19:03:31.213502+0000","last_undegraded":"2023-07-14T19:03:31.213502+0000","last_fullsized":"2023-07-14T19:03:31.213502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:26.819356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":72,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.677712+0000","last_change":"2023-07-14T19:02:29.938459+0000","last_active":"2023-07-14T19:03:31.677712+0000","last_peered":"2023-07-14T19:03:31.677712+0000","last_clean":"2023-07-14T19:03:31.677712+0000","last_became_active":"2023-07-14T19:02:29.938212+0000","last_became_peered":"2023-07-14T19:02:29.938212+0000","last_unstale":"2023-07-14T19:03:31.677712+0000","last_undegraded":"2023-07-14T19:03:31.677712+0000","last_fullsized":"2023-07-14T19:03:31.677712+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:34.199818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":43,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281471+0000","last_change":"2023-07-14T19:02:30.586345+0000","last_active":"2023-07-14T19:03:31.281471+0000","last_peered":"2023-07-14T19:03:31.281471+0000","last_clean":"2023-07-14T19:03:31.281471+0000","last_became_active":"2023-07-14T19:02:30.585949+0000","last_became_peered":"2023-07-14T19:02:30.585949+0000","last_unstale":"2023-07-14T19:03:31.281471+0000","last_undegraded":"2023-07-14T19:03:31.281471+0000","last_fullsized":"2023-07-14T19:03:31.281471+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:50:32.754337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210309+0000","last_change":"2023-07-14T19:02:34.582922+0000","last_active":"2023-07-14T19:03:31.210309+0000","last_peered":"2023-07-14T19:03:31.210309+0000","last_clean":"2023-07-14T19:03:31.210309+0000","last_became_active":"2023-07-14T19:02:34.582146+0000","last_became_peered":"2023-07-14T19:02:34.582146+0000","last_unstale":"2023-07-14T19:03:31.210309+0000","last_undegraded":"2023-07-14T19:03:31.210309+0000","last_fullsized":"2023-07-14T19:03:31.210309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:15.121038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212755+0000","last_change":"2023-07-14T19:02:29.938215+0000","last_active":"2023-07-14T19:03:31.212755+0000","last_peered":"2023-07-14T19:03:31.212755+0000","last_clean":"2023-07-14T19:03:31.212755+0000","last_became_active":"2023-07-14T19:02:29.937889+0000","last_became_peered":"2023-07-14T19:02:29.937889+0000","last_unstale":"2023-07-14T19:03:31.212755+0000","last_undegraded":"2023-07-14T19:03:31.212755+0000","last_fullsized":"2023-07-14T19:03:31.212755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:19.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208284+0000","last_change":"2023-07-14T19:02:30.569622+0000","last_active":"2023-07-14T19:03:31.208284+0000","last_peered":"2023-07-14T19:03:31.208284+0000","last_clean":"2023-07-14T19:03:31.208284+0000","last_became_active":"2023-07-14T19:02:30.569201+0000","last_became_peered":"2023-07-14T19:02:30.569201+0000","last_unstale":"2023-07-14T19:03:31.208284+0000","last_undegraded":"2023-07-14T19:03:31.208284+0000","last_fullsized":"2023-07-14T19:03:31.208284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:53:07.451914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214242+0000","last_change":"2023-07-14T19:02:34.580978+0000","last_active":"2023-07-14T19:03:31.214242+0000","last_peered":"2023-07-14T19:03:31.214242+0000","last_clean":"2023-07-14T19:03:31.214242+0000","last_became_active":"2023-07-14T19:02:34.580503+0000","last_became_peered":"2023-07-14T19:02:34.580503+0000","last_unstale":"2023-07-14T19:03:31.214242+0000","last_undegraded":"2023-07-14T19:03:31.214242+0000","last_fullsized":"2023-07-14T19:03:31.214242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:06.819824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219409+0000","last_change":"2023-07-14T19:02:29.942400+0000","last_active":"2023-07-14T19:03:31.219409+0000","last_peered":"2023-07-14T19:03:31.219409+0000","last_clean":"2023-07-14T19:03:31.219409+0000","last_became_active":"2023-07-14T19:02:29.942080+0000","last_became_peered":"2023-07-14T19:02:29.942080+0000","last_unstale":"2023-07-14T19:03:31.219409+0000","last_undegraded":"2023-07-14T19:03:31.219409+0000","last_fullsized":"2023-07-14T19:03:31.219409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:09:06.130256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281477+0000","last_change":"2023-07-14T19:02:30.582869+0000","last_active":"2023-07-14T19:03:31.281477+0000","last_peered":"2023-07-14T19:03:31.281477+0000","last_clean":"2023-07-14T19:03:31.281477+0000","last_became_active":"2023-07-14T19:02:30.582558+0000","last_became_peered":"2023-07-14T19:02:30.582558+0000","last_unstale":"2023-07-14T19:03:31.281477+0000","last_undegraded":"2023-07-14T19:03:31.281477+0000","last_fullsized":"2023-07-14T19:03:31.281477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:51:36.705425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.216738+0000","last_change":"2023-07-14T19:03:31.216738+0000","last_active":"2023-07-14T19:03:31.216738+0000","last_peered":"2023-07-14T19:03:31.216738+0000","last_clean":"2023-07-14T19:03:31.216738+0000","last_became_active":"2023-07-14T19:03:31.216310+0000","last_became_peered":"2023-07-14T19:03:31.216310+0000","last_unstale":"2023-07-14T19:03:31.216738+0000","last_undegraded":"2023-07-14T19:03:31.216738+0000","last_fullsized":"2023-07-14T19:03:31.216738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:26.359586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'30","reported_seq":149,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:32.686954+0000","last_change":"2023-07-14T19:02:29.939314+0000","last_active":"2023-07-14T19:03:32.686954+0000","last_peered":"2023-07-14T19:03:32.686954+0000","last_clean":"2023-07-14T19:03:32.686954+0000","last_became_active":"2023-07-14T19:02:29.939000+0000","last_became_peered":"2023-07-14T19:02:29.939000+0000","last_unstale":"2023-07-14T19:03:32.686954+0000","last_undegraded":"2023-07-14T19:03:32.686954+0000","last_fullsized":"2023-07-14T19:03:32.686954+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:43.443151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56970,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":93,"num_read_kb":102,"num_write":31,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210508+0000","last_change":"2023-07-14T19:02:30.572052+0000","last_active":"2023-07-14T19:03:31.210508+0000","last_peered":"2023-07-14T19:03:31.210508+0000","last_clean":"2023-07-14T19:03:31.210508+0000","last_became_active":"2023-07-14T19:02:30.571739+0000","last_became_peered":"2023-07-14T19:02:30.571739+0000","last_unstale":"2023-07-14T19:03:31.210508+0000","last_undegraded":"2023-07-14T19:03:31.210508+0000","last_fullsized":"2023-07-14T19:03:31.210508+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:54:15.509239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219888+0000","last_change":"2023-07-14T19:02:34.576663+0000","last_active":"2023-07-14T19:03:31.219888+0000","last_peered":"2023-07-14T19:03:31.219888+0000","last_clean":"2023-07-14T19:03:31.219888+0000","last_became_active":"2023-07-14T19:02:34.576162+0000","last_became_peered":"2023-07-14T19:02:34.576162+0000","last_unstale":"2023-07-14T19:03:31.219888+0000","last_undegraded":"2023-07-14T19:03:31.219888+0000","last_fullsized":"2023-07-14T19:03:31.219888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:50.099226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.225143+0000","last_change":"2023-07-14T19:03:31.225143+0000","last_active":"2023-07-14T19:03:31.225143+0000","last_peered":"2023-07-14T19:03:31.225143+0000","last_clean":"2023-07-14T19:03:31.225143+0000","last_became_active":"2023-07-14T19:03:31.215377+0000","last_became_peered":"2023-07-14T19:03:31.215377+0000","last_unstale":"2023-07-14T19:03:31.225143+0000","last_undegraded":"2023-07-14T19:03:31.225143+0000","last_fullsized":"2023-07-14T19:03:31.225143+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:23.915375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209950+0000","last_change":"2023-07-14T19:02:30.573798+0000","last_active":"2023-07-14T19:03:31.209950+0000","last_peered":"2023-07-14T19:03:31.209950+0000","last_clean":"2023-07-14T19:03:31.209950+0000","last_became_active":"2023-07-14T19:02:30.572568+0000","last_became_peered":"2023-07-14T19:02:30.572568+0000","last_unstale":"2023-07-14T19:03:31.209950+0000","last_undegraded":"2023-07-14T19:03:31.209950+0000","last_fullsized":"2023-07-14T19:03:31.209950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:50.422090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210143+0000","last_change":"2023-07-14T19:02:34.582256+0000","last_active":"2023-07-14T19:03:31.210143+0000","last_peered":"2023-07-14T19:03:31.210143+0000","last_clean":"2023-07-14T19:03:31.210143+0000","last_became_active":"2023-07-14T19:02:34.581799+0000","last_became_peered":"2023-07-14T19:02:34.581799+0000","last_unstale":"2023-07-14T19:03:31.210143+0000","last_undegraded":"2023-07-14T19:03:31.210143+0000","last_fullsized":"2023-07-14T19:03:31.210143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:26.965376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219545+0000","last_change":"2023-07-14T19:02:29.943874+0000","last_active":"2023-07-14T19:03:31.219545+0000","last_peered":"2023-07-14T19:03:31.219545+0000","last_clean":"2023-07-14T19:03:31.219545+0000","last_became_active":"2023-07-14T19:02:29.943596+0000","last_became_peered":"2023-07-14T19:02:29.943596+0000","last_unstale":"2023-07-14T19:03:31.219545+0000","last_undegraded":"2023-07-14T19:03:31.219545+0000","last_fullsized":"2023-07-14T19:03:31.219545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:17.878050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211874+0000","last_change":"2023-07-14T19:02:30.572157+0000","last_active":"2023-07-14T19:03:31.211874+0000","last_peered":"2023-07-14T19:03:31.211874+0000","last_clean":"2023-07-14T19:03:31.211874+0000","last_became_active":"2023-07-14T19:02:30.571877+0000","last_became_peered":"2023-07-14T19:02:30.571877+0000","last_unstale":"2023-07-14T19:03:31.211874+0000","last_undegraded":"2023-07-14T19:03:31.211874+0000","last_fullsized":"2023-07-14T19:03:31.211874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:10:27.779325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210374+0000","last_change":"2023-07-14T19:02:34.588594+0000","last_active":"2023-07-14T19:03:31.210374+0000","last_peered":"2023-07-14T19:03:31.210374+0000","last_clean":"2023-07-14T19:03:31.210374+0000","last_became_active":"2023-07-14T19:02:34.587763+0000","last_became_peered":"2023-07-14T19:02:34.587763+0000","last_unstale":"2023-07-14T19:03:31.210374+0000","last_undegraded":"2023-07-14T19:03:31.210374+0000","last_fullsized":"2023-07-14T19:03:31.210374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:02.399855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212239+0000","last_change":"2023-07-14T19:02:29.939145+0000","last_active":"2023-07-14T19:03:31.212239+0000","last_peered":"2023-07-14T19:03:31.212239+0000","last_clean":"2023-07-14T19:03:31.212239+0000","last_became_active":"2023-07-14T19:02:29.938816+0000","last_became_peered":"2023-07-14T19:02:29.938816+0000","last_unstale":"2023-07-14T19:03:31.212239+0000","last_undegraded":"2023-07-14T19:03:31.212239+0000","last_fullsized":"2023-07-14T19:03:31.212239+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:05.245627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":46,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213189+0000","last_change":"2023-07-14T19:02:30.572694+0000","last_active":"2023-07-14T19:03:31.213189+0000","last_peered":"2023-07-14T19:03:31.213189+0000","last_clean":"2023-07-14T19:03:31.213189+0000","last_became_active":"2023-07-14T19:02:30.572391+0000","last_became_peered":"2023-07-14T19:02:30.572391+0000","last_unstale":"2023-07-14T19:03:31.213189+0000","last_undegraded":"2023-07-14T19:03:31.213189+0000","last_fullsized":"2023-07-14T19:03:31.213189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:38:18.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214930+0000","last_change":"2023-07-14T19:03:31.214930+0000","last_active":"2023-07-14T19:03:31.214930+0000","last_peered":"2023-07-14T19:03:31.214930+0000","last_clean":"2023-07-14T19:03:31.214930+0000","last_became_active":"2023-07-14T19:03:31.214574+0000","last_became_peered":"2023-07-14T19:03:31.214574+0000","last_unstale":"2023-07-14T19:03:31.214930+0000","last_undegraded":"2023-07-14T19:03:31.214930+0000","last_fullsized":"2023-07-14T19:03:31.214930+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:17.395228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.236699+0000","last_change":"2023-07-14T19:03:31.236699+0000","last_active":"2023-07-14T19:03:31.236699+0000","last_peered":"2023-07-14T19:03:31.236699+0000","last_clean":"2023-07-14T19:03:31.236699+0000","last_became_active":"2023-07-14T19:03:31.216096+0000","last_became_peered":"2023-07-14T19:03:31.216096+0000","last_unstale":"2023-07-14T19:03:31.236699+0000","last_undegraded":"2023-07-14T19:03:31.236699+0000","last_fullsized":"2023-07-14T19:03:31.236699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:23:44.113371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.282151+0000","last_change":"2023-07-14T19:03:31.282151+0000","last_active":"2023-07-14T19:03:31.282151+0000","last_peered":"2023-07-14T19:03:31.282151+0000","last_clean":"2023-07-14T19:03:31.282151+0000","last_became_active":"2023-07-14T19:03:31.281763+0000","last_became_peered":"2023-07-14T19:03:31.281763+0000","last_unstale":"2023-07-14T19:03:31.282151+0000","last_undegraded":"2023-07-14T19:03:31.282151+0000","last_fullsized":"2023-07-14T19:03:31.282151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:13.429632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218880+0000","last_change":"2023-07-14T19:02:34.584170+0000","last_active":"2023-07-14T19:03:31.218880+0000","last_peered":"2023-07-14T19:03:31.218880+0000","last_clean":"2023-07-14T19:03:31.218880+0000","last_became_active":"2023-07-14T19:02:34.583773+0000","last_became_peered":"2023-07-14T19:02:34.583773+0000","last_unstale":"2023-07-14T19:03:31.218880+0000","last_undegraded":"2023-07-14T19:03:31.218880+0000","last_fullsized":"2023-07-14T19:03:31.218880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:32:31.049038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210253+0000","last_change":"2023-07-14T19:02:29.938386+0000","last_active":"2023-07-14T19:03:31.210253+0000","last_peered":"2023-07-14T19:03:31.210253+0000","last_clean":"2023-07-14T19:03:31.210253+0000","last_became_active":"2023-07-14T19:02:29.937394+0000","last_became_peered":"2023-07-14T19:02:29.937394+0000","last_unstale":"2023-07-14T19:03:31.210253+0000","last_undegraded":"2023-07-14T19:03:31.210253+0000","last_fullsized":"2023-07-14T19:03:31.210253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:44:18.573548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210241+0000","last_change":"2023-07-14T19:02:30.572485+0000","last_active":"2023-07-14T19:03:31.210241+0000","last_peered":"2023-07-14T19:03:31.210241+0000","last_clean":"2023-07-14T19:03:31.210241+0000","last_became_active":"2023-07-14T19:02:30.572177+0000","last_became_peered":"2023-07-14T19:02:30.572177+0000","last_unstale":"2023-07-14T19:03:31.210241+0000","last_undegraded":"2023-07-14T19:03:31.210241+0000","last_fullsized":"2023-07-14T19:03:31.210241+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:00:15.417585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210735+0000","last_change":"2023-07-14T19:02:34.582340+0000","last_active":"2023-07-14T19:03:31.210735+0000","last_peered":"2023-07-14T19:03:31.210735+0000","last_clean":"2023-07-14T19:03:31.210735+0000","last_became_active":"2023-07-14T19:02:34.581936+0000","last_became_peered":"2023-07-14T19:02:34.581936+0000","last_unstale":"2023-07-14T19:03:31.210735+0000","last_undegraded":"2023-07-14T19:03:31.210735+0000","last_fullsized":"2023-07-14T19:03:31.210735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:00.136660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210852+0000","last_change":"2023-07-14T19:02:29.941404+0000","last_active":"2023-07-14T19:03:31.210852+0000","last_peered":"2023-07-14T19:03:31.210852+0000","last_clean":"2023-07-14T19:03:31.210852+0000","last_became_active":"2023-07-14T19:02:29.941052+0000","last_became_peered":"2023-07-14T19:02:29.941052+0000","last_unstale":"2023-07-14T19:03:31.210852+0000","last_undegraded":"2023-07-14T19:03:31.210852+0000","last_fullsized":"2023-07-14T19:03:31.210852+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:23:27.513794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214237+0000","last_change":"2023-07-14T19:02:30.565671+0000","last_active":"2023-07-14T19:03:31.214237+0000","last_peered":"2023-07-14T19:03:31.214237+0000","last_clean":"2023-07-14T19:03:31.214237+0000","last_became_active":"2023-07-14T19:02:30.565375+0000","last_became_peered":"2023-07-14T19:02:30.565375+0000","last_unstale":"2023-07-14T19:03:31.214237+0000","last_undegraded":"2023-07-14T19:03:31.214237+0000","last_fullsized":"2023-07-14T19:03:31.214237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:12:48.571954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210193+0000","last_change":"2023-07-14T19:02:34.577692+0000","last_active":"2023-07-14T19:03:31.210193+0000","last_peered":"2023-07-14T19:03:31.210193+0000","last_clean":"2023-07-14T19:03:31.210193+0000","last_became_active":"2023-07-14T19:02:34.577329+0000","last_became_peered":"2023-07-14T19:02:34.577329+0000","last_unstale":"2023-07-14T19:03:31.210193+0000","last_undegraded":"2023-07-14T19:03:31.210193+0000","last_fullsized":"2023-07-14T19:03:31.210193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:07.060258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212127+0000","last_change":"2023-07-14T19:02:29.938330+0000","last_active":"2023-07-14T19:03:31.212127+0000","last_peered":"2023-07-14T19:03:31.212127+0000","last_clean":"2023-07-14T19:03:31.212127+0000","last_became_active":"2023-07-14T19:02:29.938022+0000","last_became_peered":"2023-07-14T19:02:29.938022+0000","last_unstale":"2023-07-14T19:03:31.212127+0000","last_undegraded":"2023-07-14T19:03:31.212127+0000","last_fullsized":"2023-07-14T19:03:31.212127+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:24.520411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213923+0000","last_change":"2023-07-14T19:02:30.569353+0000","last_active":"2023-07-14T19:03:31.213923+0000","last_peered":"2023-07-14T19:03:31.213923+0000","last_clean":"2023-07-14T19:03:31.213923+0000","last_became_active":"2023-07-14T19:02:30.569037+0000","last_became_peered":"2023-07-14T19:02:30.569037+0000","last_unstale":"2023-07-14T19:03:31.213923+0000","last_undegraded":"2023-07-14T19:03:31.213923+0000","last_fullsized":"2023-07-14T19:03:31.213923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:38.723738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210218+0000","last_change":"2023-07-14T19:02:34.580786+0000","last_active":"2023-07-14T19:03:31.210218+0000","last_peered":"2023-07-14T19:03:31.210218+0000","last_clean":"2023-07-14T19:03:31.210218+0000","last_became_active":"2023-07-14T19:02:34.580395+0000","last_became_peered":"2023-07-14T19:02:34.580395+0000","last_unstale":"2023-07-14T19:03:31.210218+0000","last_undegraded":"2023-07-14T19:03:31.210218+0000","last_fullsized":"2023-07-14T19:03:31.210218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:45:16.010542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218850+0000","last_change":"2023-07-14T19:02:30.574302+0000","last_active":"2023-07-14T19:03:31.218850+0000","last_peered":"2023-07-14T19:03:31.218850+0000","last_clean":"2023-07-14T19:03:31.218850+0000","last_became_active":"2023-07-14T19:02:30.573912+0000","last_became_peered":"2023-07-14T19:02:30.573912+0000","last_unstale":"2023-07-14T19:03:31.218850+0000","last_undegraded":"2023-07-14T19:03:31.218850+0000","last_fullsized":"2023-07-14T19:03:31.218850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:57.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":75,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:33.680390+0000","last_change":"2023-07-14T19:02:29.942302+0000","last_active":"2023-07-14T19:03:33.680390+0000","last_peered":"2023-07-14T19:03:33.680390+0000","last_clean":"2023-07-14T19:03:33.680390+0000","last_became_active":"2023-07-14T19:02:29.942005+0000","last_became_peered":"2023-07-14T19:02:29.942005+0000","last_unstale":"2023-07-14T19:03:33.680390+0000","last_undegraded":"2023-07-14T19:03:33.680390+0000","last_fullsized":"2023-07-14T19:03:33.680390+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:32.223130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":45,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212188+0000","last_change":"2023-07-14T19:02:34.578248+0000","last_active":"2023-07-14T19:03:31.212188+0000","last_peered":"2023-07-14T19:03:31.212188+0000","last_clean":"2023-07-14T19:03:31.212188+0000","last_became_active":"2023-07-14T19:02:34.577930+0000","last_became_peered":"2023-07-14T19:02:34.577930+0000","last_unstale":"2023-07-14T19:03:31.212188+0000","last_undegraded":"2023-07-14T19:03:31.212188+0000","last_fullsized":"2023-07-14T19:03:31.212188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:13.074222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213462+0000","last_change":"2023-07-14T19:02:30.582892+0000","last_active":"2023-07-14T19:03:31.213462+0000","last_peered":"2023-07-14T19:03:31.213462+0000","last_clean":"2023-07-14T19:03:31.213462+0000","last_became_active":"2023-07-14T19:02:30.582588+0000","last_became_peered":"2023-07-14T19:02:30.582588+0000","last_unstale":"2023-07-14T19:03:31.213462+0000","last_undegraded":"2023-07-14T19:03:31.213462+0000","last_fullsized":"2023-07-14T19:03:31.213462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:06.056870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214203+0000","last_change":"2023-07-14T19:02:29.935857+0000","last_active":"2023-07-14T19:03:31.214203+0000","last_peered":"2023-07-14T19:03:31.214203+0000","last_clean":"2023-07-14T19:03:31.214203+0000","last_became_active":"2023-07-14T19:02:29.935489+0000","last_became_peered":"2023-07-14T19:02:29.935489+0000","last_unstale":"2023-07-14T19:03:31.214203+0000","last_undegraded":"2023-07-14T19:03:31.214203+0000","last_fullsized":"2023-07-14T19:03:31.214203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:08.956235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212941+0000","last_change":"2023-07-14T19:01:31.967476+0000","last_active":"2023-07-14T19:03:31.212941+0000","last_peered":"2023-07-14T19:03:31.212941+0000","last_clean":"2023-07-14T19:03:31.212941+0000","last_became_active":"2023-07-14T19:01:31.967146+0000","last_became_peered":"2023-07-14T19:01:31.967146+0000","last_unstale":"2023-07-14T19:03:31.212941+0000","last_undegraded":"2023-07-14T19:03:31.212941+0000","last_fullsized":"2023-07-14T19:03:31.212941+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_clean_scrub_stamp":"2023-07-14T18:58:40.634620+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:56.440561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219439+0000","last_change":"2023-07-14T19:02:34.571823+0000","last_active":"2023-07-14T19:03:31.219439+0000","last_peered":"2023-07-14T19:03:31.219439+0000","last_clean":"2023-07-14T19:03:31.219439+0000","last_became_active":"2023-07-14T19:02:34.571359+0000","last_became_peered":"2023-07-14T19:02:34.571359+0000","last_unstale":"2023-07-14T19:03:31.219439+0000","last_undegraded":"2023-07-14T19:03:31.219439+0000","last_fullsized":"2023-07-14T19:03:31.219439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:28.423706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209714+0000","last_change":"2023-07-14T19:02:30.573736+0000","last_active":"2023-07-14T19:03:31.209714+0000","last_peered":"2023-07-14T19:03:31.209714+0000","last_clean":"2023-07-14T19:03:31.209714+0000","last_became_active":"2023-07-14T19:02:30.573381+0000","last_became_peered":"2023-07-14T19:02:30.573381+0000","last_unstale":"2023-07-14T19:03:31.209714+0000","last_undegraded":"2023-07-14T19:03:31.209714+0000","last_fullsized":"2023-07-14T19:03:31.209714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:09.104343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213067+0000","last_change":"2023-07-14T19:02:29.937660+0000","last_active":"2023-07-14T19:03:31.213067+0000","last_peered":"2023-07-14T19:03:31.213067+0000","last_clean":"2023-07-14T19:03:31.213067+0000","last_became_active":"2023-07-14T19:02:29.937348+0000","last_became_peered":"2023-07-14T19:02:29.937348+0000","last_unstale":"2023-07-14T19:03:31.213067+0000","last_undegraded":"2023-07-14T19:03:31.213067+0000","last_fullsized":"2023-07-14T19:03:31.213067+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:55:06.823754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211142+0000","last_change":"2023-07-14T19:02:34.581534+0000","last_active":"2023-07-14T19:03:31.211142+0000","last_peered":"2023-07-14T19:03:31.211142+0000","last_clean":"2023-07-14T19:03:31.211142+0000","last_became_active":"2023-07-14T19:02:34.580998+0000","last_became_peered":"2023-07-14T19:02:34.580998+0000","last_unstale":"2023-07-14T19:03:31.211142+0000","last_undegraded":"2023-07-14T19:03:31.211142+0000","last_fullsized":"2023-07-14T19:03:31.211142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:05:30.391146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210585+0000","last_change":"2023-07-14T19:02:29.942336+0000","last_active":"2023-07-14T19:03:31.210585+0000","last_peered":"2023-07-14T19:03:31.210585+0000","last_clean":"2023-07-14T19:03:31.210585+0000","last_became_active":"2023-07-14T19:02:29.942033+0000","last_became_peered":"2023-07-14T19:02:29.942033+0000","last_unstale":"2023-07-14T19:03:31.210585+0000","last_undegraded":"2023-07-14T19:03:31.210585+0000","last_fullsized":"2023-07-14T19:03:31.210585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:04:44.877772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210377+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:03:31.210377+0000","last_peered":"2023-07-14T19:03:31.210377+0000","last_clean":"2023-07-14T19:03:31.210377+0000","last_became_active":"2023-07-14T19:02:30.571129+0000","last_became_peered":"2023-07-14T19:02:30.571129+0000","last_unstale":"2023-07-14T19:03:31.210377+0000","last_undegraded":"2023-07-14T19:03:31.210377+0000","last_fullsized":"2023-07-14T19:03:31.210377+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:38:56.156291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213158+0000","last_change":"2023-07-14T19:02:34.580740+0000","last_active":"2023-07-14T19:03:31.213158+0000","last_peered":"2023-07-14T19:03:31.213158+0000","last_clean":"2023-07-14T19:03:31.213158+0000","last_became_active":"2023-07-14T19:02:34.580344+0000","last_became_peered":"2023-07-14T19:02:34.580344+0000","last_unstale":"2023-07-14T19:03:31.213158+0000","last_undegraded":"2023-07-14T19:03:31.213158+0000","last_fullsized":"2023-07-14T19:03:31.213158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:33:02.155617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'5","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210352+0000","last_change":"2023-07-14T19:02:29.941349+0000","last_active":"2023-07-14T19:03:31.210352+0000","last_peered":"2023-07-14T19:03:31.210352+0000","last_clean":"2023-07-14T19:03:31.210352+0000","last_became_active":"2023-07-14T19:02:29.941039+0000","last_became_peered":"2023-07-14T19:02:29.941039+0000","last_unstale":"2023-07-14T19:03:31.210352+0000","last_undegraded":"2023-07-14T19:03:31.210352+0000","last_fullsized":"2023-07-14T19:03:31.210352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:38.367807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214004+0000","last_change":"2023-07-14T19:02:30.577546+0000","last_active":"2023-07-14T19:03:31.214004+0000","last_peered":"2023-07-14T19:03:31.214004+0000","last_clean":"2023-07-14T19:03:31.214004+0000","last_became_active":"2023-07-14T19:02:30.576833+0000","last_became_peered":"2023-07-14T19:02:30.576833+0000","last_unstale":"2023-07-14T19:03:31.214004+0000","last_undegraded":"2023-07-14T19:03:31.214004+0000","last_fullsized":"2023-07-14T19:03:31.214004+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:00:55.318416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214192+0000","last_change":"2023-07-14T19:02:34.580655+0000","last_active":"2023-07-14T19:03:31.214192+0000","last_peered":"2023-07-14T19:03:31.214192+0000","last_clean":"2023-07-14T19:03:31.214192+0000","last_became_active":"2023-07-14T19:02:34.580186+0000","last_became_peered":"2023-07-14T19:02:34.580186+0000","last_unstale":"2023-07-14T19:03:31.214192+0000","last_undegraded":"2023-07-14T19:03:31.214192+0000","last_fullsized":"2023-07-14T19:03:31.214192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:17:17.836503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212620+0000","last_change":"2023-07-14T19:02:29.938343+0000","last_active":"2023-07-14T19:03:31.212620+0000","last_peered":"2023-07-14T19:03:31.212620+0000","last_clean":"2023-07-14T19:03:31.212620+0000","last_became_active":"2023-07-14T19:02:29.938053+0000","last_became_peered":"2023-07-14T19:02:29.938053+0000","last_unstale":"2023-07-14T19:03:31.212620+0000","last_undegraded":"2023-07-14T19:03:31.212620+0000","last_fullsized":"2023-07-14T19:03:31.212620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:50:45.655040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208419+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:03:31.208419+0000","last_peered":"2023-07-14T19:03:31.208419+0000","last_clean":"2023-07-14T19:03:31.208419+0000","last_became_active":"2023-07-14T19:02:30.572436+0000","last_became_peered":"2023-07-14T19:02:30.572436+0000","last_unstale":"2023-07-14T19:03:31.208419+0000","last_undegraded":"2023-07-14T19:03:31.208419+0000","last_fullsized":"2023-07-14T19:03:31.208419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:11:10.005214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213237+0000","last_change":"2023-07-14T19:02:34.582900+0000","last_active":"2023-07-14T19:03:31.213237+0000","last_peered":"2023-07-14T19:03:31.213237+0000","last_clean":"2023-07-14T19:03:31.213237+0000","last_became_active":"2023-07-14T19:02:34.582493+0000","last_became_peered":"2023-07-14T19:02:34.582493+0000","last_unstale":"2023-07-14T19:03:31.213237+0000","last_undegraded":"2023-07-14T19:03:31.213237+0000","last_fullsized":"2023-07-14T19:03:31.213237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:00:45.030002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.280975+0000","last_change":"2023-07-14T19:02:29.941819+0000","last_active":"2023-07-14T19:03:31.280975+0000","last_peered":"2023-07-14T19:03:31.280975+0000","last_clean":"2023-07-14T19:03:31.280975+0000","last_became_active":"2023-07-14T19:02:29.941469+0000","last_became_peered":"2023-07-14T19:02:29.941469+0000","last_unstale":"2023-07-14T19:03:31.280975+0000","last_undegraded":"2023-07-14T19:03:31.280975+0000","last_fullsized":"2023-07-14T19:03:31.280975+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:28.479332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211086+0000","last_change":"2023-07-14T19:02:30.565125+0000","last_active":"2023-07-14T19:03:31.211086+0000","last_peered":"2023-07-14T19:03:31.211086+0000","last_clean":"2023-07-14T19:03:31.211086+0000","last_became_active":"2023-07-14T19:02:30.564820+0000","last_became_peered":"2023-07-14T19:02:30.564820+0000","last_unstale":"2023-07-14T19:03:31.211086+0000","last_undegraded":"2023-07-14T19:03:31.211086+0000","last_fullsized":"2023-07-14T19:03:31.211086+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:16:52.054472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.221013+0000","last_change":"2023-07-14T19:03:31.220582+0000","last_active":"2023-07-14T19:03:31.221013+0000","last_peered":"2023-07-14T19:03:31.221013+0000","last_clean":"2023-07-14T19:03:31.221013+0000","last_became_active":"2023-07-14T19:03:31.220162+0000","last_became_peered":"2023-07-14T19:03:31.220162+0000","last_unstale":"2023-07-14T19:03:31.221013+0000","last_undegraded":"2023-07-14T19:03:31.221013+0000","last_fullsized":"2023-07-14T19:03:31.221013+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:00.068361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213644+0000","last_change":"2023-07-14T19:02:29.938027+0000","last_active":"2023-07-14T19:03:31.213644+0000","last_peered":"2023-07-14T19:03:31.213644+0000","last_clean":"2023-07-14T19:03:31.213644+0000","last_became_active":"2023-07-14T19:02:29.937690+0000","last_became_peered":"2023-07-14T19:02:29.937690+0000","last_unstale":"2023-07-14T19:03:31.213644+0000","last_undegraded":"2023-07-14T19:03:31.213644+0000","last_fullsized":"2023-07-14T19:03:31.213644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:21.192935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210779+0000","last_change":"2023-07-14T19:02:30.576369+0000","last_active":"2023-07-14T19:03:31.210779+0000","last_peered":"2023-07-14T19:03:31.210779+0000","last_clean":"2023-07-14T19:03:31.210779+0000","last_became_active":"2023-07-14T19:02:30.576113+0000","last_became_peered":"2023-07-14T19:02:30.576113+0000","last_unstale":"2023-07-14T19:03:31.210779+0000","last_undegraded":"2023-07-14T19:03:31.210779+0000","last_fullsized":"2023-07-14T19:03:31.210779+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:57.280886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213432+0000","last_change":"2023-07-14T19:02:34.583724+0000","last_active":"2023-07-14T19:03:31.213432+0000","last_peered":"2023-07-14T19:03:31.213432+0000","last_clean":"2023-07-14T19:03:31.213432+0000","last_became_active":"2023-07-14T19:02:34.583218+0000","last_became_peered":"2023-07-14T19:02:34.583218+0000","last_unstale":"2023-07-14T19:03:31.213432+0000","last_undegraded":"2023-07-14T19:03:31.213432+0000","last_fullsized":"2023-07-14T19:03:31.213432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:28:38.807293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213965+0000","last_change":"2023-07-14T19:02:29.938908+0000","last_active":"2023-07-14T19:03:31.213965+0000","last_peered":"2023-07-14T19:03:31.213965+0000","last_clean":"2023-07-14T19:03:31.213965+0000","last_became_active":"2023-07-14T19:02:29.938650+0000","last_became_peered":"2023-07-14T19:02:29.938650+0000","last_unstale":"2023-07-14T19:03:31.213965+0000","last_undegraded":"2023-07-14T19:03:31.213965+0000","last_fullsized":"2023-07-14T19:03:31.213965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:22:03.037314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213600+0000","last_change":"2023-07-14T19:02:30.571870+0000","last_active":"2023-07-14T19:03:31.213600+0000","last_peered":"2023-07-14T19:03:31.213600+0000","last_clean":"2023-07-14T19:03:31.213600+0000","last_became_active":"2023-07-14T19:02:30.571521+0000","last_became_peered":"2023-07-14T19:02:30.571521+0000","last_unstale":"2023-07-14T19:03:31.213600+0000","last_undegraded":"2023-07-14T19:03:31.213600+0000","last_fullsized":"2023-07-14T19:03:31.213600+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:47.125420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213270+0000","last_change":"2023-07-14T19:02:34.589900+0000","last_active":"2023-07-14T19:03:31.213270+0000","last_peered":"2023-07-14T19:03:31.213270+0000","last_clean":"2023-07-14T19:03:31.213270+0000","last_became_active":"2023-07-14T19:02:34.589481+0000","last_became_peered":"2023-07-14T19:02:34.589481+0000","last_unstale":"2023-07-14T19:03:31.213270+0000","last_undegraded":"2023-07-14T19:03:31.213270+0000","last_fullsized":"2023-07-14T19:03:31.213270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:07.853098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212669+0000","last_change":"2023-07-14T19:02:29.940315+0000","last_active":"2023-07-14T19:03:31.212669+0000","last_peered":"2023-07-14T19:03:31.212669+0000","last_clean":"2023-07-14T19:03:31.212669+0000","last_became_active":"2023-07-14T19:02:29.939991+0000","last_became_peered":"2023-07-14T19:02:29.939991+0000","last_unstale":"2023-07-14T19:03:31.212669+0000","last_undegraded":"2023-07-14T19:03:31.212669+0000","last_fullsized":"2023-07-14T19:03:31.212669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:30.521480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212329+0000","last_change":"2023-07-14T19:02:30.583959+0000","last_active":"2023-07-14T19:03:31.212329+0000","last_peered":"2023-07-14T19:03:31.212329+0000","last_clean":"2023-07-14T19:03:31.212329+0000","last_became_active":"2023-07-14T19:02:30.583657+0000","last_became_peered":"2023-07-14T19:02:30.583657+0000","last_unstale":"2023-07-14T19:03:31.212329+0000","last_undegraded":"2023-07-14T19:03:31.212329+0000","last_fullsized":"2023-07-14T19:03:31.212329+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:04:07.282087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281103+0000","last_change":"2023-07-14T19:02:34.579328+0000","last_active":"2023-07-14T19:03:31.281103+0000","last_peered":"2023-07-14T19:03:31.281103+0000","last_clean":"2023-07-14T19:03:31.281103+0000","last_became_active":"2023-07-14T19:02:34.578904+0000","last_became_peered":"2023-07-14T19:02:34.578904+0000","last_unstale":"2023-07-14T19:03:31.281103+0000","last_undegraded":"2023-07-14T19:03:31.281103+0000","last_fullsized":"2023-07-14T19:03:31.281103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:15.218624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214589+0000","last_change":"2023-07-14T19:02:29.940972+0000","last_active":"2023-07-14T19:03:31.214589+0000","last_peered":"2023-07-14T19:03:31.214589+0000","last_clean":"2023-07-14T19:03:31.214589+0000","last_became_active":"2023-07-14T19:02:29.940670+0000","last_became_peered":"2023-07-14T19:02:29.940670+0000","last_unstale":"2023-07-14T19:03:31.214589+0000","last_undegraded":"2023-07-14T19:03:31.214589+0000","last_fullsized":"2023-07-14T19:03:31.214589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:55:01.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212697+0000","last_change":"2023-07-14T19:02:30.583714+0000","last_active":"2023-07-14T19:03:31.212697+0000","last_peered":"2023-07-14T19:03:31.212697+0000","last_clean":"2023-07-14T19:03:31.212697+0000","last_became_active":"2023-07-14T19:02:30.583404+0000","last_became_peered":"2023-07-14T19:02:30.583404+0000","last_unstale":"2023-07-14T19:03:31.212697+0000","last_undegraded":"2023-07-14T19:03:31.212697+0000","last_fullsized":"2023-07-14T19:03:31.212697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:55.354431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207943+0000","last_change":"2023-07-14T19:02:34.582985+0000","last_active":"2023-07-14T19:03:31.207943+0000","last_peered":"2023-07-14T19:03:31.207943+0000","last_clean":"2023-07-14T19:03:31.207943+0000","last_became_active":"2023-07-14T19:02:34.582606+0000","last_became_peered":"2023-07-14T19:02:34.582606+0000","last_unstale":"2023-07-14T19:03:31.207943+0000","last_undegraded":"2023-07-14T19:03:31.207943+0000","last_fullsized":"2023-07-14T19:03:31.207943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:55:50.958196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219531+0000","last_change":"2023-07-14T19:02:29.944064+0000","last_active":"2023-07-14T19:03:31.219531+0000","last_peered":"2023-07-14T19:03:31.219531+0000","last_clean":"2023-07-14T19:03:31.219531+0000","last_became_active":"2023-07-14T19:02:29.943770+0000","last_became_peered":"2023-07-14T19:02:29.943770+0000","last_unstale":"2023-07-14T19:03:31.219531+0000","last_undegraded":"2023-07-14T19:03:31.219531+0000","last_fullsized":"2023-07-14T19:03:31.219531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:02:39.670163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213041+0000","last_change":"2023-07-14T19:02:30.580668+0000","last_active":"2023-07-14T19:03:31.213041+0000","last_peered":"2023-07-14T19:03:31.213041+0000","last_clean":"2023-07-14T19:03:31.213041+0000","last_became_active":"2023-07-14T19:02:30.580297+0000","last_became_peered":"2023-07-14T19:02:30.580297+0000","last_unstale":"2023-07-14T19:03:31.213041+0000","last_undegraded":"2023-07-14T19:03:31.213041+0000","last_fullsized":"2023-07-14T19:03:31.213041+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:03:37.240639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.217879+0000","last_change":"2023-07-14T19:02:34.578894+0000","last_active":"2023-07-14T19:03:31.217879+0000","last_peered":"2023-07-14T19:03:31.217879+0000","last_clean":"2023-07-14T19:03:31.217879+0000","last_became_active":"2023-07-14T19:02:34.578463+0000","last_became_peered":"2023-07-14T19:02:34.578463+0000","last_unstale":"2023-07-14T19:03:31.217879+0000","last_undegraded":"2023-07-14T19:03:31.217879+0000","last_fullsized":"2023-07-14T19:03:31.217879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:53.046945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210215+0000","last_change":"2023-07-14T19:02:29.938430+0000","last_active":"2023-07-14T19:03:31.210215+0000","last_peered":"2023-07-14T19:03:31.210215+0000","last_clean":"2023-07-14T19:03:31.210215+0000","last_became_active":"2023-07-14T19:02:29.938118+0000","last_became_peered":"2023-07-14T19:02:29.938118+0000","last_unstale":"2023-07-14T19:03:31.210215+0000","last_undegraded":"2023-07-14T19:03:31.210215+0000","last_fullsized":"2023-07-14T19:03:31.210215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:59.720559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212504+0000","last_change":"2023-07-14T19:02:30.575124+0000","last_active":"2023-07-14T19:03:31.212504+0000","last_peered":"2023-07-14T19:03:31.212504+0000","last_clean":"2023-07-14T19:03:31.212504+0000","last_became_active":"2023-07-14T19:02:30.574805+0000","last_became_peered":"2023-07-14T19:02:30.574805+0000","last_unstale":"2023-07-14T19:03:31.212504+0000","last_undegraded":"2023-07-14T19:03:31.212504+0000","last_fullsized":"2023-07-14T19:03:31.212504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:42:48.292682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210923+0000","last_change":"2023-07-14T19:02:30.573897+0000","last_active":"2023-07-14T19:03:31.210923+0000","last_peered":"2023-07-14T19:03:31.210923+0000","last_clean":"2023-07-14T19:03:31.210923+0000","last_became_active":"2023-07-14T19:02:30.573585+0000","last_became_peered":"2023-07-14T19:02:30.573585+0000","last_unstale":"2023-07-14T19:03:31.210923+0000","last_undegraded":"2023-07-14T19:03:31.210923+0000","last_fullsized":"2023-07-14T19:03:31.210923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:26:13.564561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215202+0000","last_change":"2023-07-14T19:02:30.572390+0000","last_active":"2023-07-14T19:03:31.215202+0000","last_peered":"2023-07-14T19:03:31.215202+0000","last_clean":"2023-07-14T19:03:31.215202+0000","last_became_active":"2023-07-14T19:02:30.572093+0000","last_became_peered":"2023-07-14T19:02:30.572093+0000","last_unstale":"2023-07-14T19:03:31.215202+0000","last_undegraded":"2023-07-14T19:03:31.215202+0000","last_fullsized":"2023-07-14T19:03:31.215202+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:57:10.086878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213816+0000","last_change":"2023-07-14T19:02:30.572548+0000","last_active":"2023-07-14T19:03:31.213816+0000","last_peered":"2023-07-14T19:03:31.213816+0000","last_clean":"2023-07-14T19:03:31.213816+0000","last_became_active":"2023-07-14T19:02:30.572260+0000","last_became_peered":"2023-07-14T19:02:30.572260+0000","last_unstale":"2023-07-14T19:03:31.213816+0000","last_undegraded":"2023-07-14T19:03:31.213816+0000","last_fullsized":"2023-07-14T19:03:31.213816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:55.025634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208138+0000","last_change":"2023-07-14T19:02:30.569856+0000","last_active":"2023-07-14T19:03:31.208138+0000","last_peered":"2023-07-14T19:03:31.208138+0000","last_clean":"2023-07-14T19:03:31.208138+0000","last_became_active":"2023-07-14T19:02:30.569546+0000","last_became_peered":"2023-07-14T19:02:30.569546+0000","last_unstale":"2023-07-14T19:03:31.208138+0000","last_undegraded":"2023-07-14T19:03:31.208138+0000","last_fullsized":"2023-07-14T19:03:31.208138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:43.152189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210350+0000","last_change":"2023-07-14T19:02:30.570836+0000","last_active":"2023-07-14T19:03:31.210350+0000","last_peered":"2023-07-14T19:03:31.210350+0000","last_clean":"2023-07-14T19:03:31.210350+0000","last_became_active":"2023-07-14T19:02:30.569675+0000","last_became_peered":"2023-07-14T19:02:30.569675+0000","last_unstale":"2023-07-14T19:03:31.210350+0000","last_undegraded":"2023-07-14T19:03:31.210350+0000","last_fullsized":"2023-07-14T19:03:31.210350+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:26:19.766316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209916+0000","last_change":"2023-07-14T19:02:30.566846+0000","last_active":"2023-07-14T19:03:31.209916+0000","last_peered":"2023-07-14T19:03:31.209916+0000","last_clean":"2023-07-14T19:03:31.209916+0000","last_became_active":"2023-07-14T19:02:30.566525+0000","last_became_peered":"2023-07-14T19:02:30.566525+0000","last_unstale":"2023-07-14T19:03:31.209916+0000","last_undegraded":"2023-07-14T19:03:31.209916+0000","last_fullsized":"2023-07-14T19:03:31.209916+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:12:37.744089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215127+0000","last_change":"2023-07-14T19:02:30.579680+0000","last_active":"2023-07-14T19:03:31.215127+0000","last_peered":"2023-07-14T19:03:31.215127+0000","last_clean":"2023-07-14T19:03:31.215127+0000","last_became_active":"2023-07-14T19:02:30.579379+0000","last_became_peered":"2023-07-14T19:02:30.579379+0000","last_unstale":"2023-07-14T19:03:31.215127+0000","last_undegraded":"2023-07-14T19:03:31.215127+0000","last_fullsized":"2023-07-14T19:03:31.215127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:00:25.682516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214145+0000","last_change":"2023-07-14T19:02:30.577722+0000","last_active":"2023-07-14T19:03:31.214145+0000","last_peered":"2023-07-14T19:03:31.214145+0000","last_clean":"2023-07-14T19:03:31.214145+0000","last_became_active":"2023-07-14T19:02:30.577247+0000","last_became_peered":"2023-07-14T19:02:30.577247+0000","last_unstale":"2023-07-14T19:03:31.214145+0000","last_undegraded":"2023-07-14T19:03:31.214145+0000","last_fullsized":"2023-07-14T19:03:31.214145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:19:14.172473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218436+0000","last_change":"2023-07-14T19:02:30.571052+0000","last_active":"2023-07-14T19:03:31.218436+0000","last_peered":"2023-07-14T19:03:31.218436+0000","last_clean":"2023-07-14T19:03:31.218436+0000","last_became_active":"2023-07-14T19:02:30.570722+0000","last_became_peered":"2023-07-14T19:02:30.570722+0000","last_unstale":"2023-07-14T19:03:31.218436+0000","last_undegraded":"2023-07-14T19:03:31.218436+0000","last_fullsized":"2023-07-14T19:03:31.218436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:50:37.718571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213772+0000","last_change":"2023-07-14T19:02:30.575685+0000","last_active":"2023-07-14T19:03:31.213772+0000","last_peered":"2023-07-14T19:03:31.213772+0000","last_clean":"2023-07-14T19:03:31.213772+0000","last_became_active":"2023-07-14T19:02:30.575379+0000","last_became_peered":"2023-07-14T19:02:30.575379+0000","last_unstale":"2023-07-14T19:03:31.213772+0000","last_undegraded":"2023-07-14T19:03:31.213772+0000","last_fullsized":"2023-07-14T19:03:31.213772+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:14:01.199281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281456+0000","last_change":"2023-07-14T19:02:30.582450+0000","last_active":"2023-07-14T19:03:31.281456+0000","last_peered":"2023-07-14T19:03:31.281456+0000","last_clean":"2023-07-14T19:03:31.281456+0000","last_became_active":"2023-07-14T19:02:30.582074+0000","last_became_peered":"2023-07-14T19:02:30.582074+0000","last_unstale":"2023-07-14T19:03:31.281456+0000","last_undegraded":"2023-07-14T19:03:31.281456+0000","last_fullsized":"2023-07-14T19:03:31.281456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:50.575652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210265+0000","last_change":"2023-07-14T19:02:30.573042+0000","last_active":"2023-07-14T19:03:31.210265+0000","last_peered":"2023-07-14T19:03:31.210265+0000","last_clean":"2023-07-14T19:03:31.210265+0000","last_became_active":"2023-07-14T19:02:30.572498+0000","last_became_peered":"2023-07-14T19:02:30.572498+0000","last_unstale":"2023-07-14T19:03:31.210265+0000","last_undegraded":"2023-07-14T19:03:31.210265+0000","last_fullsized":"2023-07-14T19:03:31.210265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:01.947296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210278+0000","last_change":"2023-07-14T19:02:30.569441+0000","last_active":"2023-07-14T19:03:31.210278+0000","last_peered":"2023-07-14T19:03:31.210278+0000","last_clean":"2023-07-14T19:03:31.210278+0000","last_became_active":"2023-07-14T19:02:30.569180+0000","last_became_peered":"2023-07-14T19:02:30.569180+0000","last_unstale":"2023-07-14T19:03:31.210278+0000","last_undegraded":"2023-07-14T19:03:31.210278+0000","last_fullsized":"2023-07-14T19:03:31.210278+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:24.866934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61264,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":284,"num_read_kb":209,"num_write":153,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":146,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514521,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":476824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.52200000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.63}]},{"osd":2,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.53900000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.61799999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.60299999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.67200000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.68400000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.73999999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.72899999999999998}]}]},{"osd":10,"up_from":69,"seq":296352743452,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1654784,"data_stored":1121258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.439},{"interface":"front","average":{"1min":0.504,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.36399999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.63100000000000001,"15min":0.621},"min":{"1min":0.48499999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.57299999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.442,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45400000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79000000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51100000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59999999999999998,"15min":0.60099999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.41699999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.47099999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.56399999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.61399999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69499999999999995,"15min":0.69299999999999995},"min":{"1min":0.51200000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89100000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.60099999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77800000000000002,"15min":0.77900000000000003},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96699999999999997,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.627}]},{"osd":9,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77800000000000002,"15min":0.78200000000000003},"min":{"1min":0.59599999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.747,"15min":0.753},"min":{"1min":0.57199999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88,"5min":1.226,"15min":1.226},"last":0.58999999999999997}]},{"osd":11,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.66700000000000004}]}]},{"osd":9,"up_from":64,"seq":274877906976,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":481010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.42099999999999999,"5min":0.63,"15min":0.63},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.38300000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79000000000000004,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.71299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52900000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47699999999999998,"15min":0.49299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66300000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.30199999999999999},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.34200000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.41099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72499999999999998,"5min":1.008,"15min":1.008},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.746,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.63}]},{"osd":5,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70499999999999996,"5min":1.026,"15min":1.026},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.70099999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.62},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.69999999999999996,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.81799999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.73499999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.73699999999999999,"15min":0.755},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81399999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84299999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.59499999999999997}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.746,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.70399999999999996,"15min":0.71499999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79400000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.745}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]}]},{"osd":8,"up_from":57,"seq":244813135909,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":476391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88300000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.436},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.32600000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90900000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76300000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.36399999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.878,"15min":0.878},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.495}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.68400000000000005,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.47399999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.626,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.53000000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.503},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.94299999999999995,"5min":1.01,"15min":1.01},"last":0.626}]},{"osd":7,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.871,"15min":0.871},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0369999999999999,"5min":1.046,"15min":1.046},"last":0.65300000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.755,"15min":0.75},"min":{"1min":0.53200000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.57199999999999995}]},{"osd":10,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77600000000000002,"15min":0.76600000000000001},"min":{"1min":0.64600000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.016,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.67800000000000005},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.70299999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0700000000000001,"5min":1.081,"15min":1.081},"last":0.61799999999999999}]},{"osd":11,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.56499999999999995}]}]},{"osd":7,"up_from":50,"seq":214748364842,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1576960,"data_stored":1072763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56999999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.318},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.40899999999999997},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.35799999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57399999999999995,"15min":0.54900000000000004},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.82299999999999995},{"interface":"front","average":{"1min":0.622,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.56000000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71699999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.71399999999999997},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.53200000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.47699999999999998,"15min":0.45900000000000002},"min":{"1min":0.38300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.442},{"interface":"front","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.48599999999999999},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69999999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.50900000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.627,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.87,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.68500000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.63},"min":{"1min":0.54100000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94899999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.57999999999999996}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.78700000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74099999999999999,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96999999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.997,"5min":0.76300000000000001,"15min":0.72399999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":6.8970000000000002,"5min":6.8970000000000002,"15min":6.8970000000000002},"last":0.75}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.72299999999999998},"min":{"1min":0.60599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95299999999999996,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.745,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.64700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.624}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.77200000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528366,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":472090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39900000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.54700000000000004,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41999999999999998,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56200000000000006,"5min":1.591,"15min":1.591},"last":0.27600000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.55700000000000005},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83699999999999997,"5min":1.159,"15min":1.159},"last":0.433},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":1.49,"15min":1.49},"last":0.42399999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73799999999999999,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61099999999999999,"15min":0.59299999999999997},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.77900000000000003,"5min":1.419,"15min":1.419},"last":0.80300000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.34300000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.48199999999999998,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":2.077,"15min":2.077},"last":0.45600000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85299999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64600000000000002,"15min":0.624},"min":{"1min":0.45500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80900000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.68999999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65100000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76200000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.59599999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.66800000000000004,"15min":0.67900000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76300000000000001,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.373},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.75,"15min":0.73099999999999998},"min":{"1min":0.51500000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.86699999999999999,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.65000000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73599999999999999},"min":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.68000000000000005}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.77400000000000002,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78900000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.48099999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82499999999999996,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.55700000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.88600000000000001,"5min":1.234,"15min":1.234},"last":0.747}]},{"osd":11,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]}]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":528970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52400000000000002,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.3370000000000002},"last":0.378},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.379,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":0.35599999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.55200000000000005,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.52500000000000002,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.50900000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88300000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81999999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.49399999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.44600000000000001,"15min":0.39100000000000001},"min":{"1min":0.36599999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44600000000000001,"15min":0.40200000000000002},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.39700000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.54800000000000004}]},{"osd":6,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61799999999999999,"15min":0.59199999999999997},"min":{"1min":0.53800000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.433},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.65000000000000002,"15min":0.60199999999999998},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.3209999999999997,"5min":4.3209999999999997,"15min":4.3209999999999997},"last":0.53800000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.54900000000000004,"15min":0.51700000000000002},"min":{"1min":0.439,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.628,"15min":0.58199999999999996},"min":{"1min":0.52200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.52200000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72899999999999998,"15min":0.71599999999999997},"min":{"1min":0.498,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.99299999999999999,"5min":0.996,"15min":0.996},"last":0.57099999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.79000000000000004,"15min":0.77300000000000002},"min":{"1min":0.54800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.60099999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.66000000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.77000000000000002,"15min":0.76000000000000001},"min":{"1min":0.625,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.625},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.73199999999999998,"15min":0.72499999999999998},"min":{"1min":0.60699999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.90300000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.60699999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.63600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":463639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.159,"15min":0.159},"max":{"1min":0.72499999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.48999999999999999},"min":{"1min":0.40300000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.75800000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.65900000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.37,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.746,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.41299999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:01:07 2023","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.34899999999999998,"15min":0.34000000000000002},"min":{"1min":0.28699999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.435,"5min":0.35999999999999999,"15min":0.34799999999999998},"min":{"1min":0.29499999999999998,"5min":0.251,"15min":0.251},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.32700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.372,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68700000000000006,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.627},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.50900000000000001},"min":{"1min":0.43099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.73899999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.46000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91700000000000004,"5min":1.042,"15min":1.042},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.58999999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.496,"15min":0.495},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72099999999999997,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.51600000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.72299999999999998,"5min":0.67800000000000005,"15min":0.67000000000000004},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.1799999999999999,"5min":1.1799999999999999,"15min":1.1799999999999999},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.65500000000000003,"15min":0.65700000000000003},"min":{"1min":0.46700000000000003,"5min":0.37,"15min":0.37},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.60199999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70199999999999996,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.024,"15min":1.024},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.57999999999999996}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.78200000000000003,"5min":1.204,"15min":1.204},"last":0.503},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72499999999999998,"15min":0.746},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82199999999999995,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.63900000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.877,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.79700000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.71499999999999997}]},{"osd":11,"last update":"Fri Jul 14 19:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.66900000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":530628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78700000000000003,"5min":1.111,"15min":1.111},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.39000000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.628,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.67200000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77000000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55200000000000005,"15min":0.53900000000000003},"min":{"1min":0.45500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78100000000000003,"5min":1.046,"15min":1.046},"last":0.40799999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.68500000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.44500000000000001,"15min":0.41799999999999998},"min":{"1min":0.33600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.65800000000000003}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.59599999999999997,"15min":0.56599999999999995},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.59999999999999998},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60299999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89300000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.438}]},{"osd":7,"last update":"Fri Jul 14 19:02:29 2023","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60199999999999998,"15min":0.59499999999999997},"min":{"1min":0.45500000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.67100000000000004,"5min":0.61599999999999999,"15min":0.60699999999999998},"min":{"1min":0.47499999999999998,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.64900000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.443,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.59599999999999997,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.748}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92600000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72799999999999998,"15min":0.71999999999999997},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.76000000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.753,"15min":0.73099999999999998},"min":{"1min":0.69799999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.752,"15min":0.72699999999999998},"min":{"1min":0.68700000000000006,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.79600000000000004}]},{"osd":11,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.81000000000000005},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.77100000000000002}]}]},{"osd":3,"up_from":25,"seq":107374182458,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":472284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.188,"15min":0.188},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35299999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.215,"15min":0.215},"max":{"1min":0.52800000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.314}]},{"osd":1,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71799999999999997,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.49199999999999999}]},{"osd":2,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.39600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.745,"5min":4.6509999999999998,"15min":4.6509999999999998},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60799999999999998,"15min":0.66400000000000003},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.45600000000000002}]},{"osd":4,"last update":"Fri Jul 14 19:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.503}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.621,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.498,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.52400000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.82799999999999996,"5min":1.119,"15min":1.119},"last":0.437}]},{"osd":7,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.376,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.61299999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66200000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76000000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.55900000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.68899999999999995,"15min":0.69699999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91900000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69799999999999995,"15min":0.70399999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.629}]},{"osd":11,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.60199999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920822,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":476809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.503,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.38100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38400000000000001,"15min":0.34799999999999998},"min":{"1min":0.33600000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.59199999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.44},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40100000000000002,"15min":0.36499999999999999},"min":{"1min":0.34399999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.71799999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.376}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.89500000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.58499999999999996,"15min":0.54300000000000004},"min":{"1min":0.45500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.105,"15min":1.105},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55300000000000005,"15min":0.51200000000000001},"min":{"1min":0.432,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72299999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54900000000000004,"15min":0.50600000000000001},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.48199999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57599999999999996,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.69599999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.60399999999999998},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":1.038,"15min":1.038},"last":0.59099999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.442,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.873,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.73599999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68600000000000005,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.71499999999999997},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.75600000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.58399999999999996,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.79600000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:02:33 2023","interfaces":[{"interface":"back","average":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"min":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.77600000000000002}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.81000000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1067951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46200000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.495},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.41399999999999998,"15min":0.38900000000000001},"min":{"1min":0.309,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.43099999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70599999999999996,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.54500000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.57599999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60199999999999998,"15min":0.58799999999999997},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84099999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64500000000000002,"15min":0.63100000000000001},"min":{"1min":0.40200000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.60699999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63900000000000001,"15min":0.622},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.65100000000000002,"15min":0.64000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.64700000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.71699999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.753},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.66300000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:02:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.755},{"interface":"front","average":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":0.69199999999999995},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.73399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:03:38.399 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph status --format=json 2023-07-14T19:03:38.950 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:39.437 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:39 smithi078 ceph-mon[127425]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:39.438 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:39 smithi078 ceph-mon[127425]: pgmap v291: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 18 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:39.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:39 smithi102 ceph-mon[129280]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:39.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:39 smithi102 ceph-mon[129280]: pgmap v291: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 18 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:39.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:39 smithi088 ceph-mon[128973]: from='client.15108 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:39.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:39 smithi088 ceph-mon[128973]: pgmap v291: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 4 op/s; 18 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:40.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3923088676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df410a2d0 msgr2=0x7f2df410c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3923088676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df410a2d0 0x7f2df410c6f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f2dec00b0a0 tx=0x7f2dec00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3923088676 shutdown_connections 2023-07-14T19:03:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3923088676 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df410a2d0 0x7f2df410c6f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3923088676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df4100ef0 0x7f2df41012c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3923088676 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2df4101800 0x7f2df4109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3923088676 >> 172.21.15.78:0/3923088676 conn(0x7f2df4075d30 msgr2=0x7f2df4076130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3923088676 shutdown_connections 2023-07-14T19:03:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.128+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3923088676 wait complete. 2023-07-14T19:03:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfc50d700 1 Processor -- start 2023-07-14T19:03:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfc50d700 1 -- start start 2023-07-14T19:03:40.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfc50d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ef0 0x7f2df4195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfc50d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df4101800 0x7f2df4196310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfc50d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2df410a2d0 0x7f2df419a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfc50d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df419ad90 con 0x7f2df4100ef0 2023-07-14T19:03:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfc50d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df419af00 con 0x7f2df410a2d0 2023-07-14T19:03:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfc50d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2df419b0c0 con 0x7f2df4101800 2023-07-14T19:03:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfa2a9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ef0 0x7f2df4195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:40.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfa2a9700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ef0 0x7f2df4195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35484/0 (socket says 172.21.15.78:35484) 2023-07-14T19:03:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfa2a9700 1 -- 172.21.15.78:0/3687246118 learned_addr learned my addr 172.21.15.78:0/3687246118 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2df9aa8700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df4101800 0x7f2df4196310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.129+0000 7f2dfaaaa700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2df410a2d0 0x7f2df419a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2dfa2a9700 1 -- 172.21.15.78:0/3687246118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df4101800 msgr2=0x7f2df4196310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2dfa2a9700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df4101800 0x7f2df4196310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2dfa2a9700 1 -- 172.21.15.78:0/3687246118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2df410a2d0 msgr2=0x7f2df419a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2dfa2a9700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2df410a2d0 0x7f2df419a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2dfa2a9700 1 -- 172.21.15.78:0/3687246118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dec009d00 con 0x7f2df4100ef0 2023-07-14T19:03:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2dfa2a9700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ef0 0x7f2df4195dd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2df000e9a0 tx=0x7f2df000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2deb7fe700 1 -- 172.21.15.78:0/3687246118 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0009790 con 0x7f2df4100ef0 2023-07-14T19:03:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2deb7fe700 1 -- 172.21.15.78:0/3687246118 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2df0010430 con 0x7f2df4100ef0 2023-07-14T19:03:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2df419b3a0 con 0x7f2df4100ef0 2023-07-14T19:03:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2deb7fe700 1 -- 172.21.15.78:0/3687246118 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df000cea0 con 0x7f2df4100ef0 2023-07-14T19:03:40.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.130+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2df40fec50 con 0x7f2df4100ef0 2023-07-14T19:03:40.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.131+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2df4104b30 con 0x7f2df4100ef0 2023-07-14T19:03:40.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.132+0000 7f2deb7fe700 1 -- 172.21.15.78:0/3687246118 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2df00105a0 con 0x7f2df4100ef0 2023-07-14T19:03:40.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.133+0000 7f2deb7fe700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2de00a11e0 0x7f2de00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:40.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.133+0000 7f2df9aa8700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2de00a11e0 0x7f2de00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:40.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.133+0000 7f2deb7fe700 1 -- 172.21.15.78:0/3687246118 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2df0014070 con 0x7f2df4100ef0 2023-07-14T19:03:40.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.133+0000 7f2df9aa8700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2de00a11e0 0x7f2de00a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2df41973f0 tx=0x7f2de4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:40.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.136+0000 7f2deb7fe700 1 -- 172.21.15.78:0/3687246118 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2df0096460 con 0x7f2df4100ef0 2023-07-14T19:03:40.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.397+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f2df404e340 con 0x7f2df4100ef0 2023-07-14T19:03:40.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.397+0000 7f2deb7fe700 1 -- 172.21.15.78:0/3687246118 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2315 (secure 0 0 0) 0x7f2df0026920 con 0x7f2df4100ef0 2023-07-14T19:03:40.402 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:40.402 INFO:teuthology.orchestra.run.smithi078.stdout:{"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":377,"monmap":{"epoch":3,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":87,"num_osds":12,"num_up_osds":12,"osd_up_since":1689361309,"num_in_osds":12,"osd_in_since":1689361291,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":129}],"num_pgs":129,"num_pools":4,"num_objects":64,"data_bytes":651878,"bytes_used":345800704,"bytes_avail":1151594627072,"bytes_total":1151940427776,"recovering_objects_per_sec":1,"recovering_bytes_per_sec":14,"recovering_keys_per_sec":2,"num_objects_recovered":15,"num_bytes_recovered":146,"num_keys_recovered":20,"read_bytes_sec":2991,"read_op_per_sec":5,"write_op_per_sec":0},"fsmap":{"epoch":23,"id":1,"up":3,"in":3,"max":3,"by_rank":[{"filesystem_id":1,"rank":2,"name":"f","status":"up:standby-replay","gid":24457},{"filesystem_id":1,"rank":1,"name":"e","status":"up:standby-replay","gid":24464},{"filesystem_id":1,"rank":0,"name":"c","status":"up:standby-replay","gid":24469},{"filesystem_id":1,"rank":2,"name":"h","status":"up:active","gid":24485},{"filesystem_id":1,"rank":1,"name":"i","status":"up:active","gid":24487},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24503}],"up:standby":6},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.78:8443/"}},"servicemap":{"epoch":25,"modified":"2023-07-14T19:03:36.126473+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"e":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"f":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"h":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"i":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-07-14T19:03:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2de00a11e0 msgr2=0x7f2de00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2de00a11e0 0x7f2de00a3690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2df41973f0 tx=0x7f2de4009260 comp rx=0 tx=0).stop 2023-07-14T19:03:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ef0 msgr2=0x7f2df4195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:40.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ef0 0x7f2df4195dd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f2df000e9a0 tx=0x7f2df000ecb0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 shutdown_connections 2023-07-14T19:03:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2de00a11e0 0x7f2de00a3690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2df410a2d0 0x7f2df419a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2df4100ef0 0x7f2df4195dd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 --2- 172.21.15.78:0/3687246118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2df4101800 0x7f2df4196310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:40.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 >> 172.21.15.78:0/3687246118 conn(0x7f2df4075d30 msgr2=0x7f2df4101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 shutdown_connections 2023-07-14T19:03:40.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:40.400+0000 7f2dfc50d700 1 -- 172.21.15.78:0/3687246118 wait complete. 2023-07-14T19:03:40.817 INFO:tasks.cephadm.ceph_manager.ceph:making progress, resetting timeout 2023-07-14T19:03:40.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:03:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:41.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:41 smithi078 ceph-mon[127425]: pgmap v292: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:41.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:41 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3687246118' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T19:03:41.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:41 smithi102 ceph-mon[129280]: pgmap v292: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:41.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:41 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3687246118' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T19:03:41.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:41 smithi088 ceph-mon[128973]: pgmap v292: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 14 B/s, 2 keys/s, 1 objects/s recovering 2023-07-14T19:03:41.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:41 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3687246118' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2023-07-14T19:03:42.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 -- 172.21.15.78:0/3824665847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83280ff7e0 msgr2=0x7f83281037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:42.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 --2- 172.21.15.78:0/3824665847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83280ff7e0 0x7f83281037b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f83180099c0 tx=0x7f8318009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 -- 172.21.15.78:0/3824665847 shutdown_connections 2023-07-14T19:03:42.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 --2- 172.21.15.78:0/3824665847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8328103ee0 0x7f832810ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 --2- 172.21.15.78:0/3824665847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83280fee40 0x7f83280ff210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 --2- 172.21.15.78:0/3824665847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f83280ff7e0 0x7f83281037b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 -- 172.21.15.78:0/3824665847 >> 172.21.15.78:0/3824665847 conn(0x7f8328075ca0 msgr2=0x7f83280760a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:42.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 -- 172.21.15.78:0/3824665847 shutdown_connections 2023-07-14T19:03:42.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.681+0000 7f832f5e5700 1 -- 172.21.15.78:0/3824665847 wait complete. 2023-07-14T19:03:42.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832f5e5700 1 Processor -- start 2023-07-14T19:03:42.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832f5e5700 1 -- start start 2023-07-14T19:03:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832f5e5700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83280fee40 0x7f832819a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832f5e5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83280ff7e0 0x7f832819aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832f5e5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8328103ee0 0x7f8328194a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832f5e5700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83281950a0 con 0x7f8328103ee0 2023-07-14T19:03:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832f5e5700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8328195210 con 0x7f83280fee40 2023-07-14T19:03:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832f5e5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83281953d0 con 0x7f83280ff7e0 2023-07-14T19:03:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832db82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8328103ee0 0x7f8328194a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832db82700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8328103ee0 0x7f8328194a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35500/0 (socket says 172.21.15.78:35500) 2023-07-14T19:03:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832db82700 1 -- 172.21.15.78:0/642346976 learned_addr learned my addr 172.21.15.78:0/642346976 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832d381700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83280fee40 0x7f832819a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.682+0000 7f832cb80700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83280ff7e0 0x7f832819aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f832db82700 1 -- 172.21.15.78:0/642346976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83280ff7e0 msgr2=0x7f832819aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f832db82700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83280ff7e0 0x7f832819aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f832db82700 1 -- 172.21.15.78:0/642346976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83280fee40 msgr2=0x7f832819a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f832db82700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83280fee40 0x7f832819a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f832db82700 1 -- 172.21.15.78:0/642346976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83180095f0 con 0x7f8328103ee0 2023-07-14T19:03:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f832db82700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8328103ee0 0x7f8328194a30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f832000cca0 tx=0x7f832000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f831e7fc700 1 -- 172.21.15.78:0/642346976 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832000aaf0 con 0x7f8328103ee0 2023-07-14T19:03:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83281956b0 con 0x7f8328103ee0 2023-07-14T19:03:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f831e7fc700 1 -- 172.21.15.78:0/642346976 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f832000ac50 con 0x7f8328103ee0 2023-07-14T19:03:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f831e7fc700 1 -- 172.21.15.78:0/642346976 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83200115e0 con 0x7f8328103ee0 2023-07-14T19:03:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.683+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83281a19a0 con 0x7f8328103ee0 2023-07-14T19:03:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.684+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f832804e340 con 0x7f8328103ee0 2023-07-14T19:03:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.686+0000 7f831e7fc700 1 -- 172.21.15.78:0/642346976 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f8320004ad0 con 0x7f8328103ee0 2023-07-14T19:03:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.686+0000 7f831e7fc700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f83140a11e0 0x7f83140a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.686+0000 7f832d381700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f83140a11e0 0x7f83140a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.686+0000 7f831e7fc700 1 -- 172.21.15.78:0/642346976 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f83200cd2a0 con 0x7f8328103ee0 2023-07-14T19:03:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.687+0000 7f832d381700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f83140a11e0 0x7f83140a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8324009a10 tx=0x7f8324009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.690+0000 7f831e7fc700 1 -- 172.21.15.78:0/642346976 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8320096690 con 0x7f8328103ee0 2023-07-14T19:03:42.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.872+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8328195f40 con 0x7f83140a11e0 2023-07-14T19:03:42.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.884+0000 7f831e7fc700 1 -- 172.21.15.78:0/642346976 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427193 (secure 0 0 0) 0x7f8328195f40 con 0x7f83140a11e0 2023-07-14T19:03:42.891 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:42.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f83140a11e0 msgr2=0x7f83140a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f83140a11e0 0x7f83140a3690 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f8324009a10 tx=0x7f8324009190 comp rx=0 tx=0).stop 2023-07-14T19:03:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8328103ee0 msgr2=0x7f8328194a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8328103ee0 0x7f8328194a30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f832000cca0 tx=0x7f832000a060 comp rx=0 tx=0).stop 2023-07-14T19:03:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 shutdown_connections 2023-07-14T19:03:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f83140a11e0 0x7f83140a3690 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f83280fee40 0x7f832819a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8328103ee0 0x7f8328194a30 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 --2- 172.21.15.78:0/642346976 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f83280ff7e0 0x7f832819aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 >> 172.21.15.78:0/642346976 conn(0x7f8328075ca0 msgr2=0x7f83280fc530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 shutdown_connections 2023-07-14T19:03:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:42.887+0000 7f832f5e5700 1 -- 172.21.15.78:0/642346976 wait complete. 2023-07-14T19:03:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-14T19:03:43.318 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":293,"stamp":"2023-07-14T19:03:42.127900+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651878,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":450,"num_read_kb":333,"num_write":336,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1180882,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337696,"kb_used_data":13428,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604128,"statfs":{"total":1151940427776,"available":1151594627072,"internally_reserved":0,"allocated":13750272,"data_stored":7650183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":-12,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":146,"num_keys_recovered":20,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.921186"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210016+0000","last_change":"2023-07-14T19:02:30.582427+0000","last_active":"2023-07-14T19:03:31.210016+0000","last_peered":"2023-07-14T19:03:31.210016+0000","last_clean":"2023-07-14T19:03:31.210016+0000","last_became_active":"2023-07-14T19:02:30.582149+0000","last_became_peered":"2023-07-14T19:02:30.582149+0000","last_unstale":"2023-07-14T19:03:31.210016+0000","last_undegraded":"2023-07-14T19:03:31.210016+0000","last_fullsized":"2023-07-14T19:03:31.210016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:09.726556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208324+0000","last_change":"2023-07-14T19:02:30.567053+0000","last_active":"2023-07-14T19:03:31.208324+0000","last_peered":"2023-07-14T19:03:31.208324+0000","last_clean":"2023-07-14T19:03:31.208324+0000","last_became_active":"2023-07-14T19:02:30.566736+0000","last_became_peered":"2023-07-14T19:02:30.566736+0000","last_unstale":"2023-07-14T19:03:31.208324+0000","last_undegraded":"2023-07-14T19:03:31.208324+0000","last_fullsized":"2023-07-14T19:03:31.208324+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:44:23.099904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208244+0000","last_change":"2023-07-14T19:02:30.577632+0000","last_active":"2023-07-14T19:03:31.208244+0000","last_peered":"2023-07-14T19:03:31.208244+0000","last_clean":"2023-07-14T19:03:31.208244+0000","last_became_active":"2023-07-14T19:02:30.577300+0000","last_became_peered":"2023-07-14T19:02:30.577300+0000","last_unstale":"2023-07-14T19:03:31.208244+0000","last_undegraded":"2023-07-14T19:03:31.208244+0000","last_fullsized":"2023-07-14T19:03:31.208244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:15:18.165242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213866+0000","last_change":"2023-07-14T19:02:30.581405+0000","last_active":"2023-07-14T19:03:31.213866+0000","last_peered":"2023-07-14T19:03:31.213866+0000","last_clean":"2023-07-14T19:03:31.213866+0000","last_became_active":"2023-07-14T19:02:30.581113+0000","last_became_peered":"2023-07-14T19:02:30.581113+0000","last_unstale":"2023-07-14T19:03:31.213866+0000","last_undegraded":"2023-07-14T19:03:31.213866+0000","last_fullsized":"2023-07-14T19:03:31.213866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:42:53.074489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212421+0000","last_change":"2023-07-14T19:02:30.574285+0000","last_active":"2023-07-14T19:03:31.212421+0000","last_peered":"2023-07-14T19:03:31.212421+0000","last_clean":"2023-07-14T19:03:31.212421+0000","last_became_active":"2023-07-14T19:02:30.573967+0000","last_became_peered":"2023-07-14T19:02:30.573967+0000","last_unstale":"2023-07-14T19:03:31.212421+0000","last_undegraded":"2023-07-14T19:03:31.212421+0000","last_fullsized":"2023-07-14T19:03:31.212421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:39:39.079558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215131+0000","last_change":"2023-07-14T19:02:30.574135+0000","last_active":"2023-07-14T19:03:31.215131+0000","last_peered":"2023-07-14T19:03:31.215131+0000","last_clean":"2023-07-14T19:03:31.215131+0000","last_became_active":"2023-07-14T19:02:30.573821+0000","last_became_peered":"2023-07-14T19:02:30.573821+0000","last_unstale":"2023-07-14T19:03:31.215131+0000","last_undegraded":"2023-07-14T19:03:31.215131+0000","last_fullsized":"2023-07-14T19:03:31.215131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:56.064229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212782+0000","last_change":"2023-07-14T19:02:30.578051+0000","last_active":"2023-07-14T19:03:31.212782+0000","last_peered":"2023-07-14T19:03:31.212782+0000","last_clean":"2023-07-14T19:03:31.212782+0000","last_became_active":"2023-07-14T19:02:30.577700+0000","last_became_peered":"2023-07-14T19:02:30.577700+0000","last_unstale":"2023-07-14T19:03:31.212782+0000","last_undegraded":"2023-07-14T19:03:31.212782+0000","last_fullsized":"2023-07-14T19:03:31.212782+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:40:01.289258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213306+0000","last_change":"2023-07-14T19:02:30.567480+0000","last_active":"2023-07-14T19:03:31.213306+0000","last_peered":"2023-07-14T19:03:31.213306+0000","last_clean":"2023-07-14T19:03:31.213306+0000","last_became_active":"2023-07-14T19:02:30.567147+0000","last_became_peered":"2023-07-14T19:02:30.567147+0000","last_unstale":"2023-07-14T19:03:31.213306+0000","last_undegraded":"2023-07-14T19:03:31.213306+0000","last_fullsized":"2023-07-14T19:03:31.213306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:02:57.085021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281462+0000","last_change":"2023-07-14T19:02:30.581936+0000","last_active":"2023-07-14T19:03:31.281462+0000","last_peered":"2023-07-14T19:03:31.281462+0000","last_clean":"2023-07-14T19:03:31.281462+0000","last_became_active":"2023-07-14T19:02:30.579619+0000","last_became_peered":"2023-07-14T19:02:30.579619+0000","last_unstale":"2023-07-14T19:03:31.281462+0000","last_undegraded":"2023-07-14T19:03:31.281462+0000","last_fullsized":"2023-07-14T19:03:31.281462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:23.438941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218216+0000","last_change":"2023-07-14T19:03:31.218216+0000","last_active":"2023-07-14T19:03:31.218216+0000","last_peered":"2023-07-14T19:03:31.218216+0000","last_clean":"2023-07-14T19:03:31.218216+0000","last_became_active":"2023-07-14T19:03:31.217871+0000","last_became_peered":"2023-07-14T19:03:31.217871+0000","last_unstale":"2023-07-14T19:03:31.218216+0000","last_undegraded":"2023-07-14T19:03:31.218216+0000","last_fullsized":"2023-07-14T19:03:31.218216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:56:32.678369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212386+0000","last_change":"2023-07-14T19:02:30.583662+0000","last_active":"2023-07-14T19:03:31.212386+0000","last_peered":"2023-07-14T19:03:31.212386+0000","last_clean":"2023-07-14T19:03:31.212386+0000","last_became_active":"2023-07-14T19:02:30.583352+0000","last_became_peered":"2023-07-14T19:02:30.583352+0000","last_unstale":"2023-07-14T19:03:31.212386+0000","last_undegraded":"2023-07-14T19:03:31.212386+0000","last_fullsized":"2023-07-14T19:03:31.212386+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:01:13.319863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213712+0000","last_change":"2023-07-14T19:02:30.567685+0000","last_active":"2023-07-14T19:03:31.213712+0000","last_peered":"2023-07-14T19:03:31.213712+0000","last_clean":"2023-07-14T19:03:31.213712+0000","last_became_active":"2023-07-14T19:02:30.566674+0000","last_became_peered":"2023-07-14T19:02:30.566674+0000","last_unstale":"2023-07-14T19:03:31.213712+0000","last_undegraded":"2023-07-14T19:03:31.213712+0000","last_fullsized":"2023-07-14T19:03:31.213712+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:41:27.460945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210143+0000","last_change":"2023-07-14T19:02:30.558012+0000","last_active":"2023-07-14T19:03:31.210143+0000","last_peered":"2023-07-14T19:03:31.210143+0000","last_clean":"2023-07-14T19:03:31.210143+0000","last_became_active":"2023-07-14T19:02:30.557675+0000","last_became_peered":"2023-07-14T19:02:30.557675+0000","last_unstale":"2023-07-14T19:03:31.210143+0000","last_undegraded":"2023-07-14T19:03:31.210143+0000","last_fullsized":"2023-07-14T19:03:31.210143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:55:41.257017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207948+0000","last_change":"2023-07-14T19:02:30.570341+0000","last_active":"2023-07-14T19:03:31.207948+0000","last_peered":"2023-07-14T19:03:31.207948+0000","last_clean":"2023-07-14T19:03:31.207948+0000","last_became_active":"2023-07-14T19:02:30.570091+0000","last_became_peered":"2023-07-14T19:02:30.570091+0000","last_unstale":"2023-07-14T19:03:31.207948+0000","last_undegraded":"2023-07-14T19:03:31.207948+0000","last_fullsized":"2023-07-14T19:03:31.207948+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:52:13.273335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207886+0000","last_change":"2023-07-14T19:02:30.573746+0000","last_active":"2023-07-14T19:03:31.207886+0000","last_peered":"2023-07-14T19:03:31.207886+0000","last_clean":"2023-07-14T19:03:31.207886+0000","last_became_active":"2023-07-14T19:02:30.573307+0000","last_became_peered":"2023-07-14T19:02:30.573307+0000","last_unstale":"2023-07-14T19:03:31.207886+0000","last_undegraded":"2023-07-14T19:03:31.207886+0000","last_fullsized":"2023-07-14T19:03:31.207886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:49.299962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.280974+0000","last_change":"2023-07-14T19:02:30.578352+0000","last_active":"2023-07-14T19:03:31.280974+0000","last_peered":"2023-07-14T19:03:31.280974+0000","last_clean":"2023-07-14T19:03:31.280974+0000","last_became_active":"2023-07-14T19:02:30.577631+0000","last_became_peered":"2023-07-14T19:02:30.577631+0000","last_unstale":"2023-07-14T19:03:31.280974+0000","last_undegraded":"2023-07-14T19:03:31.280974+0000","last_fullsized":"2023-07-14T19:03:31.280974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:07:01.573189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209912+0000","last_change":"2023-07-14T19:02:30.577813+0000","last_active":"2023-07-14T19:03:31.209912+0000","last_peered":"2023-07-14T19:03:31.209912+0000","last_clean":"2023-07-14T19:03:31.209912+0000","last_became_active":"2023-07-14T19:02:30.577482+0000","last_became_peered":"2023-07-14T19:02:30.577482+0000","last_unstale":"2023-07-14T19:03:31.209912+0000","last_undegraded":"2023-07-14T19:03:31.209912+0000","last_fullsized":"2023-07-14T19:03:31.209912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:11:33.046274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210222+0000","last_change":"2023-07-14T19:02:30.583353+0000","last_active":"2023-07-14T19:03:31.210222+0000","last_peered":"2023-07-14T19:03:31.210222+0000","last_clean":"2023-07-14T19:03:31.210222+0000","last_became_active":"2023-07-14T19:02:30.583037+0000","last_became_peered":"2023-07-14T19:02:30.583037+0000","last_unstale":"2023-07-14T19:03:31.210222+0000","last_undegraded":"2023-07-14T19:03:31.210222+0000","last_fullsized":"2023-07-14T19:03:31.210222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:43:40.021757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214491+0000","last_change":"2023-07-14T19:02:30.569788+0000","last_active":"2023-07-14T19:03:31.214491+0000","last_peered":"2023-07-14T19:03:31.214491+0000","last_clean":"2023-07-14T19:03:31.214491+0000","last_became_active":"2023-07-14T19:02:30.569461+0000","last_became_peered":"2023-07-14T19:02:30.569461+0000","last_unstale":"2023-07-14T19:03:31.214491+0000","last_undegraded":"2023-07-14T19:03:31.214491+0000","last_fullsized":"2023-07-14T19:03:31.214491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:49.917455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214902+0000","last_change":"2023-07-14T19:02:34.585220+0000","last_active":"2023-07-14T19:03:31.214902+0000","last_peered":"2023-07-14T19:03:31.214902+0000","last_clean":"2023-07-14T19:03:31.214902+0000","last_became_active":"2023-07-14T19:02:34.584771+0000","last_became_peered":"2023-07-14T19:02:34.584771+0000","last_unstale":"2023-07-14T19:03:31.214902+0000","last_undegraded":"2023-07-14T19:03:31.214902+0000","last_fullsized":"2023-07-14T19:03:31.214902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:38.451682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210698+0000","last_change":"2023-07-14T19:02:29.938571+0000","last_active":"2023-07-14T19:03:31.210698+0000","last_peered":"2023-07-14T19:03:31.210698+0000","last_clean":"2023-07-14T19:03:31.210698+0000","last_became_active":"2023-07-14T19:02:29.938270+0000","last_became_peered":"2023-07-14T19:02:29.938270+0000","last_unstale":"2023-07-14T19:03:31.210698+0000","last_undegraded":"2023-07-14T19:03:31.210698+0000","last_fullsized":"2023-07-14T19:03:31.210698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:02.148507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214143+0000","last_change":"2023-07-14T19:02:30.575894+0000","last_active":"2023-07-14T19:03:31.214143+0000","last_peered":"2023-07-14T19:03:31.214143+0000","last_clean":"2023-07-14T19:03:31.214143+0000","last_became_active":"2023-07-14T19:02:30.575607+0000","last_became_peered":"2023-07-14T19:02:30.575607+0000","last_unstale":"2023-07-14T19:03:31.214143+0000","last_undegraded":"2023-07-14T19:03:31.214143+0000","last_fullsized":"2023-07-14T19:03:31.214143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:37.751234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218192+0000","last_change":"2023-07-14T19:03:31.218192+0000","last_active":"2023-07-14T19:03:31.218192+0000","last_peered":"2023-07-14T19:03:31.218192+0000","last_clean":"2023-07-14T19:03:31.218192+0000","last_became_active":"2023-07-14T19:03:31.217708+0000","last_became_peered":"2023-07-14T19:03:31.217708+0000","last_unstale":"2023-07-14T19:03:31.218192+0000","last_undegraded":"2023-07-14T19:03:31.218192+0000","last_fullsized":"2023-07-14T19:03:31.218192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:40.671215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214160+0000","last_change":"2023-07-14T19:02:29.944136+0000","last_active":"2023-07-14T19:03:31.214160+0000","last_peered":"2023-07-14T19:03:31.214160+0000","last_clean":"2023-07-14T19:03:31.214160+0000","last_became_active":"2023-07-14T19:02:29.943751+0000","last_became_peered":"2023-07-14T19:02:29.943751+0000","last_unstale":"2023-07-14T19:03:31.214160+0000","last_undegraded":"2023-07-14T19:03:31.214160+0000","last_fullsized":"2023-07-14T19:03:31.214160+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:43:53.346383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208280+0000","last_change":"2023-07-14T19:02:30.583156+0000","last_active":"2023-07-14T19:03:31.208280+0000","last_peered":"2023-07-14T19:03:31.208280+0000","last_clean":"2023-07-14T19:03:31.208280+0000","last_became_active":"2023-07-14T19:02:30.582794+0000","last_became_peered":"2023-07-14T19:02:30.582794+0000","last_unstale":"2023-07-14T19:03:31.208280+0000","last_undegraded":"2023-07-14T19:03:31.208280+0000","last_fullsized":"2023-07-14T19:03:31.208280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:47:47.558145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214058+0000","last_change":"2023-07-14T19:02:34.590788+0000","last_active":"2023-07-14T19:03:31.214058+0000","last_peered":"2023-07-14T19:03:31.214058+0000","last_clean":"2023-07-14T19:03:31.214058+0000","last_became_active":"2023-07-14T19:02:34.590385+0000","last_became_peered":"2023-07-14T19:02:34.590385+0000","last_unstale":"2023-07-14T19:03:31.214058+0000","last_undegraded":"2023-07-14T19:03:31.214058+0000","last_fullsized":"2023-07-14T19:03:31.214058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:35.914846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.222715+0000","last_change":"2023-07-14T19:03:31.222715+0000","last_active":"2023-07-14T19:03:31.222715+0000","last_peered":"2023-07-14T19:03:31.222715+0000","last_clean":"2023-07-14T19:03:31.222715+0000","last_became_active":"2023-07-14T19:03:31.216343+0000","last_became_peered":"2023-07-14T19:03:31.216343+0000","last_unstale":"2023-07-14T19:03:31.222715+0000","last_undegraded":"2023-07-14T19:03:31.222715+0000","last_fullsized":"2023-07-14T19:03:31.222715+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:10.973001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281375+0000","last_change":"2023-07-14T19:02:30.582746+0000","last_active":"2023-07-14T19:03:31.281375+0000","last_peered":"2023-07-14T19:03:31.281375+0000","last_clean":"2023-07-14T19:03:31.281375+0000","last_became_active":"2023-07-14T19:02:30.582426+0000","last_became_peered":"2023-07-14T19:02:30.582426+0000","last_unstale":"2023-07-14T19:03:31.281375+0000","last_undegraded":"2023-07-14T19:03:31.281375+0000","last_fullsized":"2023-07-14T19:03:31.281375+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:16:53.120645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212948+0000","last_change":"2023-07-14T19:02:34.589966+0000","last_active":"2023-07-14T19:03:31.212948+0000","last_peered":"2023-07-14T19:03:31.212948+0000","last_clean":"2023-07-14T19:03:31.212948+0000","last_became_active":"2023-07-14T19:02:34.589544+0000","last_became_peered":"2023-07-14T19:02:34.589544+0000","last_unstale":"2023-07-14T19:03:31.212948+0000","last_undegraded":"2023-07-14T19:03:31.212948+0000","last_fullsized":"2023-07-14T19:03:31.212948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:08.813030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210414+0000","last_change":"2023-07-14T19:02:29.935507+0000","last_active":"2023-07-14T19:03:31.210414+0000","last_peered":"2023-07-14T19:03:31.210414+0000","last_clean":"2023-07-14T19:03:31.210414+0000","last_became_active":"2023-07-14T19:02:29.935160+0000","last_became_peered":"2023-07-14T19:02:29.935160+0000","last_unstale":"2023-07-14T19:03:31.210414+0000","last_undegraded":"2023-07-14T19:03:31.210414+0000","last_fullsized":"2023-07-14T19:03:31.210414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:02.794267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219342+0000","last_change":"2023-07-14T19:02:30.578077+0000","last_active":"2023-07-14T19:03:31.219342+0000","last_peered":"2023-07-14T19:03:31.219342+0000","last_clean":"2023-07-14T19:03:31.219342+0000","last_became_active":"2023-07-14T19:02:30.577764+0000","last_became_peered":"2023-07-14T19:02:30.577764+0000","last_unstale":"2023-07-14T19:03:31.219342+0000","last_undegraded":"2023-07-14T19:03:31.219342+0000","last_fullsized":"2023-07-14T19:03:31.219342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:40:14.188240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214161+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:03:31.214161+0000","last_peered":"2023-07-14T19:03:31.214161+0000","last_clean":"2023-07-14T19:03:31.214161+0000","last_became_active":"2023-07-14T19:02:34.583408+0000","last_became_peered":"2023-07-14T19:02:34.583408+0000","last_unstale":"2023-07-14T19:03:31.214161+0000","last_undegraded":"2023-07-14T19:03:31.214161+0000","last_fullsized":"2023-07-14T19:03:31.214161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:10.318901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":87,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:40.690784+0000","last_change":"2023-07-14T19:02:29.943899+0000","last_active":"2023-07-14T19:03:40.690784+0000","last_peered":"2023-07-14T19:03:40.690784+0000","last_clean":"2023-07-14T19:03:40.690784+0000","last_became_active":"2023-07-14T19:02:29.943585+0000","last_became_peered":"2023-07-14T19:02:29.943585+0000","last_unstale":"2023-07-14T19:03:40.690784+0000","last_undegraded":"2023-07-14T19:03:40.690784+0000","last_fullsized":"2023-07-14T19:03:40.690784+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:29.940165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210875+0000","last_change":"2023-07-14T19:02:30.567147+0000","last_active":"2023-07-14T19:03:31.210875+0000","last_peered":"2023-07-14T19:03:31.210875+0000","last_clean":"2023-07-14T19:03:31.210875+0000","last_became_active":"2023-07-14T19:02:30.566792+0000","last_became_peered":"2023-07-14T19:02:30.566792+0000","last_unstale":"2023-07-14T19:03:31.210875+0000","last_undegraded":"2023-07-14T19:03:31.210875+0000","last_fullsized":"2023-07-14T19:03:31.210875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:00:27.150739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214266+0000","last_change":"2023-07-14T19:02:34.579561+0000","last_active":"2023-07-14T19:03:31.214266+0000","last_peered":"2023-07-14T19:03:31.214266+0000","last_clean":"2023-07-14T19:03:31.214266+0000","last_became_active":"2023-07-14T19:02:34.579099+0000","last_became_peered":"2023-07-14T19:02:34.579099+0000","last_unstale":"2023-07-14T19:03:31.214266+0000","last_undegraded":"2023-07-14T19:03:31.214266+0000","last_fullsized":"2023-07-14T19:03:31.214266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:25:32.422783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.284881+0000","last_change":"2023-07-14T19:03:31.284881+0000","last_active":"2023-07-14T19:03:31.284881+0000","last_peered":"2023-07-14T19:03:31.284881+0000","last_clean":"2023-07-14T19:03:31.284881+0000","last_became_active":"2023-07-14T19:03:31.282061+0000","last_became_peered":"2023-07-14T19:03:31.282061+0000","last_unstale":"2023-07-14T19:03:31.284881+0000","last_undegraded":"2023-07-14T19:03:31.284881+0000","last_fullsized":"2023-07-14T19:03:31.284881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:32.928291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.220085+0000","last_change":"2023-07-14T19:02:30.575925+0000","last_active":"2023-07-14T19:03:31.220085+0000","last_peered":"2023-07-14T19:03:31.220085+0000","last_clean":"2023-07-14T19:03:31.220085+0000","last_became_active":"2023-07-14T19:02:30.575583+0000","last_became_peered":"2023-07-14T19:02:30.575583+0000","last_unstale":"2023-07-14T19:03:31.220085+0000","last_undegraded":"2023-07-14T19:03:31.220085+0000","last_fullsized":"2023-07-14T19:03:31.220085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:11:02.245125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209772+0000","last_change":"2023-07-14T19:02:34.587094+0000","last_active":"2023-07-14T19:03:31.209772+0000","last_peered":"2023-07-14T19:03:31.209772+0000","last_clean":"2023-07-14T19:03:31.209772+0000","last_became_active":"2023-07-14T19:02:34.586650+0000","last_became_peered":"2023-07-14T19:02:34.586650+0000","last_unstale":"2023-07-14T19:03:31.209772+0000","last_undegraded":"2023-07-14T19:03:31.209772+0000","last_fullsized":"2023-07-14T19:03:31.209772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:57.546314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209853+0000","last_change":"2023-07-14T19:02:29.934651+0000","last_active":"2023-07-14T19:03:31.209853+0000","last_peered":"2023-07-14T19:03:31.209853+0000","last_clean":"2023-07-14T19:03:31.209853+0000","last_became_active":"2023-07-14T19:02:29.934347+0000","last_became_peered":"2023-07-14T19:02:29.934347+0000","last_unstale":"2023-07-14T19:03:31.209853+0000","last_undegraded":"2023-07-14T19:03:31.209853+0000","last_fullsized":"2023-07-14T19:03:31.209853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:53:44.964924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213264+0000","last_change":"2023-07-14T19:02:30.571680+0000","last_active":"2023-07-14T19:03:31.213264+0000","last_peered":"2023-07-14T19:03:31.213264+0000","last_clean":"2023-07-14T19:03:31.213264+0000","last_became_active":"2023-07-14T19:02:30.571367+0000","last_became_peered":"2023-07-14T19:02:30.571367+0000","last_unstale":"2023-07-14T19:03:31.213264+0000","last_undegraded":"2023-07-14T19:03:31.213264+0000","last_fullsized":"2023-07-14T19:03:31.213264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:53.039270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208272+0000","last_change":"2023-07-14T19:02:34.577642+0000","last_active":"2023-07-14T19:03:31.208272+0000","last_peered":"2023-07-14T19:03:31.208272+0000","last_clean":"2023-07-14T19:03:31.208272+0000","last_became_active":"2023-07-14T19:02:34.577232+0000","last_became_peered":"2023-07-14T19:02:34.577232+0000","last_unstale":"2023-07-14T19:03:31.208272+0000","last_undegraded":"2023-07-14T19:03:31.208272+0000","last_fullsized":"2023-07-14T19:03:31.208272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:19.952531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218302+0000","last_change":"2023-07-14T19:02:29.940688+0000","last_active":"2023-07-14T19:03:31.218302+0000","last_peered":"2023-07-14T19:03:31.218302+0000","last_clean":"2023-07-14T19:03:31.218302+0000","last_became_active":"2023-07-14T19:02:29.940444+0000","last_became_peered":"2023-07-14T19:02:29.940444+0000","last_unstale":"2023-07-14T19:03:31.218302+0000","last_undegraded":"2023-07-14T19:03:31.218302+0000","last_fullsized":"2023-07-14T19:03:31.218302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:37.769743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210635+0000","last_change":"2023-07-14T19:02:30.568127+0000","last_active":"2023-07-14T19:03:31.210635+0000","last_peered":"2023-07-14T19:03:31.210635+0000","last_clean":"2023-07-14T19:03:31.210635+0000","last_became_active":"2023-07-14T19:02:30.567811+0000","last_became_peered":"2023-07-14T19:02:30.567811+0000","last_unstale":"2023-07-14T19:03:31.210635+0000","last_undegraded":"2023-07-14T19:03:31.210635+0000","last_fullsized":"2023-07-14T19:03:31.210635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:56.545251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213502+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:03:31.213502+0000","last_peered":"2023-07-14T19:03:31.213502+0000","last_clean":"2023-07-14T19:03:31.213502+0000","last_became_active":"2023-07-14T19:02:34.583480+0000","last_became_peered":"2023-07-14T19:02:34.583480+0000","last_unstale":"2023-07-14T19:03:31.213502+0000","last_undegraded":"2023-07-14T19:03:31.213502+0000","last_fullsized":"2023-07-14T19:03:31.213502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:26.819356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":77,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:36.684897+0000","last_change":"2023-07-14T19:02:29.938459+0000","last_active":"2023-07-14T19:03:36.684897+0000","last_peered":"2023-07-14T19:03:36.684897+0000","last_clean":"2023-07-14T19:03:36.684897+0000","last_became_active":"2023-07-14T19:02:29.938212+0000","last_became_peered":"2023-07-14T19:02:29.938212+0000","last_unstale":"2023-07-14T19:03:36.684897+0000","last_undegraded":"2023-07-14T19:03:36.684897+0000","last_fullsized":"2023-07-14T19:03:36.684897+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:34.199818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":48,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281471+0000","last_change":"2023-07-14T19:02:30.586345+0000","last_active":"2023-07-14T19:03:31.281471+0000","last_peered":"2023-07-14T19:03:31.281471+0000","last_clean":"2023-07-14T19:03:31.281471+0000","last_became_active":"2023-07-14T19:02:30.585949+0000","last_became_peered":"2023-07-14T19:02:30.585949+0000","last_unstale":"2023-07-14T19:03:31.281471+0000","last_undegraded":"2023-07-14T19:03:31.281471+0000","last_fullsized":"2023-07-14T19:03:31.281471+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:50:32.754337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210309+0000","last_change":"2023-07-14T19:02:34.582922+0000","last_active":"2023-07-14T19:03:31.210309+0000","last_peered":"2023-07-14T19:03:31.210309+0000","last_clean":"2023-07-14T19:03:31.210309+0000","last_became_active":"2023-07-14T19:02:34.582146+0000","last_became_peered":"2023-07-14T19:02:34.582146+0000","last_unstale":"2023-07-14T19:03:31.210309+0000","last_undegraded":"2023-07-14T19:03:31.210309+0000","last_fullsized":"2023-07-14T19:03:31.210309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:15.121038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212755+0000","last_change":"2023-07-14T19:02:29.938215+0000","last_active":"2023-07-14T19:03:31.212755+0000","last_peered":"2023-07-14T19:03:31.212755+0000","last_clean":"2023-07-14T19:03:31.212755+0000","last_became_active":"2023-07-14T19:02:29.937889+0000","last_became_peered":"2023-07-14T19:02:29.937889+0000","last_unstale":"2023-07-14T19:03:31.212755+0000","last_undegraded":"2023-07-14T19:03:31.212755+0000","last_fullsized":"2023-07-14T19:03:31.212755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:19.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208284+0000","last_change":"2023-07-14T19:02:30.569622+0000","last_active":"2023-07-14T19:03:31.208284+0000","last_peered":"2023-07-14T19:03:31.208284+0000","last_clean":"2023-07-14T19:03:31.208284+0000","last_became_active":"2023-07-14T19:02:30.569201+0000","last_became_peered":"2023-07-14T19:02:30.569201+0000","last_unstale":"2023-07-14T19:03:31.208284+0000","last_undegraded":"2023-07-14T19:03:31.208284+0000","last_fullsized":"2023-07-14T19:03:31.208284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:53:07.451914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214242+0000","last_change":"2023-07-14T19:02:34.580978+0000","last_active":"2023-07-14T19:03:31.214242+0000","last_peered":"2023-07-14T19:03:31.214242+0000","last_clean":"2023-07-14T19:03:31.214242+0000","last_became_active":"2023-07-14T19:02:34.580503+0000","last_became_peered":"2023-07-14T19:02:34.580503+0000","last_unstale":"2023-07-14T19:03:31.214242+0000","last_undegraded":"2023-07-14T19:03:31.214242+0000","last_fullsized":"2023-07-14T19:03:31.214242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:06.819824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219409+0000","last_change":"2023-07-14T19:02:29.942400+0000","last_active":"2023-07-14T19:03:31.219409+0000","last_peered":"2023-07-14T19:03:31.219409+0000","last_clean":"2023-07-14T19:03:31.219409+0000","last_became_active":"2023-07-14T19:02:29.942080+0000","last_became_peered":"2023-07-14T19:02:29.942080+0000","last_unstale":"2023-07-14T19:03:31.219409+0000","last_undegraded":"2023-07-14T19:03:31.219409+0000","last_fullsized":"2023-07-14T19:03:31.219409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:09:06.130256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281477+0000","last_change":"2023-07-14T19:02:30.582869+0000","last_active":"2023-07-14T19:03:31.281477+0000","last_peered":"2023-07-14T19:03:31.281477+0000","last_clean":"2023-07-14T19:03:31.281477+0000","last_became_active":"2023-07-14T19:02:30.582558+0000","last_became_peered":"2023-07-14T19:02:30.582558+0000","last_unstale":"2023-07-14T19:03:31.281477+0000","last_undegraded":"2023-07-14T19:03:31.281477+0000","last_fullsized":"2023-07-14T19:03:31.281477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:51:36.705425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.216738+0000","last_change":"2023-07-14T19:03:31.216738+0000","last_active":"2023-07-14T19:03:31.216738+0000","last_peered":"2023-07-14T19:03:31.216738+0000","last_clean":"2023-07-14T19:03:31.216738+0000","last_became_active":"2023-07-14T19:03:31.216310+0000","last_became_peered":"2023-07-14T19:03:31.216310+0000","last_unstale":"2023-07-14T19:03:31.216738+0000","last_undegraded":"2023-07-14T19:03:31.216738+0000","last_fullsized":"2023-07-14T19:03:31.216738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:26.359586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'30","reported_seq":159,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:37.694034+0000","last_change":"2023-07-14T19:02:29.939314+0000","last_active":"2023-07-14T19:03:37.694034+0000","last_peered":"2023-07-14T19:03:37.694034+0000","last_clean":"2023-07-14T19:03:37.694034+0000","last_became_active":"2023-07-14T19:02:29.939000+0000","last_became_peered":"2023-07-14T19:02:29.939000+0000","last_unstale":"2023-07-14T19:03:37.694034+0000","last_undegraded":"2023-07-14T19:03:37.694034+0000","last_fullsized":"2023-07-14T19:03:37.694034+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:43.443151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56970,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":103,"num_read_kb":107,"num_write":31,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210508+0000","last_change":"2023-07-14T19:02:30.572052+0000","last_active":"2023-07-14T19:03:31.210508+0000","last_peered":"2023-07-14T19:03:31.210508+0000","last_clean":"2023-07-14T19:03:31.210508+0000","last_became_active":"2023-07-14T19:02:30.571739+0000","last_became_peered":"2023-07-14T19:02:30.571739+0000","last_unstale":"2023-07-14T19:03:31.210508+0000","last_undegraded":"2023-07-14T19:03:31.210508+0000","last_fullsized":"2023-07-14T19:03:31.210508+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:54:15.509239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219888+0000","last_change":"2023-07-14T19:02:34.576663+0000","last_active":"2023-07-14T19:03:31.219888+0000","last_peered":"2023-07-14T19:03:31.219888+0000","last_clean":"2023-07-14T19:03:31.219888+0000","last_became_active":"2023-07-14T19:02:34.576162+0000","last_became_peered":"2023-07-14T19:02:34.576162+0000","last_unstale":"2023-07-14T19:03:31.219888+0000","last_undegraded":"2023-07-14T19:03:31.219888+0000","last_fullsized":"2023-07-14T19:03:31.219888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:50.099226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.225143+0000","last_change":"2023-07-14T19:03:31.225143+0000","last_active":"2023-07-14T19:03:31.225143+0000","last_peered":"2023-07-14T19:03:31.225143+0000","last_clean":"2023-07-14T19:03:31.225143+0000","last_became_active":"2023-07-14T19:03:31.215377+0000","last_became_peered":"2023-07-14T19:03:31.215377+0000","last_unstale":"2023-07-14T19:03:31.225143+0000","last_undegraded":"2023-07-14T19:03:31.225143+0000","last_fullsized":"2023-07-14T19:03:31.225143+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:23.915375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209950+0000","last_change":"2023-07-14T19:02:30.573798+0000","last_active":"2023-07-14T19:03:31.209950+0000","last_peered":"2023-07-14T19:03:31.209950+0000","last_clean":"2023-07-14T19:03:31.209950+0000","last_became_active":"2023-07-14T19:02:30.572568+0000","last_became_peered":"2023-07-14T19:02:30.572568+0000","last_unstale":"2023-07-14T19:03:31.209950+0000","last_undegraded":"2023-07-14T19:03:31.209950+0000","last_fullsized":"2023-07-14T19:03:31.209950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:50.422090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210143+0000","last_change":"2023-07-14T19:02:34.582256+0000","last_active":"2023-07-14T19:03:31.210143+0000","last_peered":"2023-07-14T19:03:31.210143+0000","last_clean":"2023-07-14T19:03:31.210143+0000","last_became_active":"2023-07-14T19:02:34.581799+0000","last_became_peered":"2023-07-14T19:02:34.581799+0000","last_unstale":"2023-07-14T19:03:31.210143+0000","last_undegraded":"2023-07-14T19:03:31.210143+0000","last_fullsized":"2023-07-14T19:03:31.210143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:26.965376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219545+0000","last_change":"2023-07-14T19:02:29.943874+0000","last_active":"2023-07-14T19:03:31.219545+0000","last_peered":"2023-07-14T19:03:31.219545+0000","last_clean":"2023-07-14T19:03:31.219545+0000","last_became_active":"2023-07-14T19:02:29.943596+0000","last_became_peered":"2023-07-14T19:02:29.943596+0000","last_unstale":"2023-07-14T19:03:31.219545+0000","last_undegraded":"2023-07-14T19:03:31.219545+0000","last_fullsized":"2023-07-14T19:03:31.219545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:17.878050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211874+0000","last_change":"2023-07-14T19:02:30.572157+0000","last_active":"2023-07-14T19:03:31.211874+0000","last_peered":"2023-07-14T19:03:31.211874+0000","last_clean":"2023-07-14T19:03:31.211874+0000","last_became_active":"2023-07-14T19:02:30.571877+0000","last_became_peered":"2023-07-14T19:02:30.571877+0000","last_unstale":"2023-07-14T19:03:31.211874+0000","last_undegraded":"2023-07-14T19:03:31.211874+0000","last_fullsized":"2023-07-14T19:03:31.211874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:10:27.779325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210374+0000","last_change":"2023-07-14T19:02:34.588594+0000","last_active":"2023-07-14T19:03:31.210374+0000","last_peered":"2023-07-14T19:03:31.210374+0000","last_clean":"2023-07-14T19:03:31.210374+0000","last_became_active":"2023-07-14T19:02:34.587763+0000","last_became_peered":"2023-07-14T19:02:34.587763+0000","last_unstale":"2023-07-14T19:03:31.210374+0000","last_undegraded":"2023-07-14T19:03:31.210374+0000","last_fullsized":"2023-07-14T19:03:31.210374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:02.399855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:40.691310+0000","last_change":"2023-07-14T19:02:29.939145+0000","last_active":"2023-07-14T19:03:40.691310+0000","last_peered":"2023-07-14T19:03:40.691310+0000","last_clean":"2023-07-14T19:03:40.691310+0000","last_became_active":"2023-07-14T19:02:29.938816+0000","last_became_peered":"2023-07-14T19:02:29.938816+0000","last_unstale":"2023-07-14T19:03:40.691310+0000","last_undegraded":"2023-07-14T19:03:40.691310+0000","last_fullsized":"2023-07-14T19:03:40.691310+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:05.245627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":56,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213189+0000","last_change":"2023-07-14T19:02:30.572694+0000","last_active":"2023-07-14T19:03:31.213189+0000","last_peered":"2023-07-14T19:03:31.213189+0000","last_clean":"2023-07-14T19:03:31.213189+0000","last_became_active":"2023-07-14T19:02:30.572391+0000","last_became_peered":"2023-07-14T19:02:30.572391+0000","last_unstale":"2023-07-14T19:03:31.213189+0000","last_undegraded":"2023-07-14T19:03:31.213189+0000","last_fullsized":"2023-07-14T19:03:31.213189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:38:18.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214930+0000","last_change":"2023-07-14T19:03:31.214930+0000","last_active":"2023-07-14T19:03:31.214930+0000","last_peered":"2023-07-14T19:03:31.214930+0000","last_clean":"2023-07-14T19:03:31.214930+0000","last_became_active":"2023-07-14T19:03:31.214574+0000","last_became_peered":"2023-07-14T19:03:31.214574+0000","last_unstale":"2023-07-14T19:03:31.214930+0000","last_undegraded":"2023-07-14T19:03:31.214930+0000","last_fullsized":"2023-07-14T19:03:31.214930+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:17.395228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.236699+0000","last_change":"2023-07-14T19:03:31.236699+0000","last_active":"2023-07-14T19:03:31.236699+0000","last_peered":"2023-07-14T19:03:31.236699+0000","last_clean":"2023-07-14T19:03:31.236699+0000","last_became_active":"2023-07-14T19:03:31.216096+0000","last_became_peered":"2023-07-14T19:03:31.216096+0000","last_unstale":"2023-07-14T19:03:31.236699+0000","last_undegraded":"2023-07-14T19:03:31.236699+0000","last_fullsized":"2023-07-14T19:03:31.236699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:23:44.113371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.282151+0000","last_change":"2023-07-14T19:03:31.282151+0000","last_active":"2023-07-14T19:03:31.282151+0000","last_peered":"2023-07-14T19:03:31.282151+0000","last_clean":"2023-07-14T19:03:31.282151+0000","last_became_active":"2023-07-14T19:03:31.281763+0000","last_became_peered":"2023-07-14T19:03:31.281763+0000","last_unstale":"2023-07-14T19:03:31.282151+0000","last_undegraded":"2023-07-14T19:03:31.282151+0000","last_fullsized":"2023-07-14T19:03:31.282151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:13.429632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218880+0000","last_change":"2023-07-14T19:02:34.584170+0000","last_active":"2023-07-14T19:03:31.218880+0000","last_peered":"2023-07-14T19:03:31.218880+0000","last_clean":"2023-07-14T19:03:31.218880+0000","last_became_active":"2023-07-14T19:02:34.583773+0000","last_became_peered":"2023-07-14T19:02:34.583773+0000","last_unstale":"2023-07-14T19:03:31.218880+0000","last_undegraded":"2023-07-14T19:03:31.218880+0000","last_fullsized":"2023-07-14T19:03:31.218880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:32:31.049038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210253+0000","last_change":"2023-07-14T19:02:29.938386+0000","last_active":"2023-07-14T19:03:31.210253+0000","last_peered":"2023-07-14T19:03:31.210253+0000","last_clean":"2023-07-14T19:03:31.210253+0000","last_became_active":"2023-07-14T19:02:29.937394+0000","last_became_peered":"2023-07-14T19:02:29.937394+0000","last_unstale":"2023-07-14T19:03:31.210253+0000","last_undegraded":"2023-07-14T19:03:31.210253+0000","last_fullsized":"2023-07-14T19:03:31.210253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:44:18.573548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210241+0000","last_change":"2023-07-14T19:02:30.572485+0000","last_active":"2023-07-14T19:03:31.210241+0000","last_peered":"2023-07-14T19:03:31.210241+0000","last_clean":"2023-07-14T19:03:31.210241+0000","last_became_active":"2023-07-14T19:02:30.572177+0000","last_became_peered":"2023-07-14T19:02:30.572177+0000","last_unstale":"2023-07-14T19:03:31.210241+0000","last_undegraded":"2023-07-14T19:03:31.210241+0000","last_fullsized":"2023-07-14T19:03:31.210241+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:00:15.417585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210735+0000","last_change":"2023-07-14T19:02:34.582340+0000","last_active":"2023-07-14T19:03:31.210735+0000","last_peered":"2023-07-14T19:03:31.210735+0000","last_clean":"2023-07-14T19:03:31.210735+0000","last_became_active":"2023-07-14T19:02:34.581936+0000","last_became_peered":"2023-07-14T19:02:34.581936+0000","last_unstale":"2023-07-14T19:03:31.210735+0000","last_undegraded":"2023-07-14T19:03:31.210735+0000","last_fullsized":"2023-07-14T19:03:31.210735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:00.136660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210852+0000","last_change":"2023-07-14T19:02:29.941404+0000","last_active":"2023-07-14T19:03:31.210852+0000","last_peered":"2023-07-14T19:03:31.210852+0000","last_clean":"2023-07-14T19:03:31.210852+0000","last_became_active":"2023-07-14T19:02:29.941052+0000","last_became_peered":"2023-07-14T19:02:29.941052+0000","last_unstale":"2023-07-14T19:03:31.210852+0000","last_undegraded":"2023-07-14T19:03:31.210852+0000","last_fullsized":"2023-07-14T19:03:31.210852+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:23:27.513794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214237+0000","last_change":"2023-07-14T19:02:30.565671+0000","last_active":"2023-07-14T19:03:31.214237+0000","last_peered":"2023-07-14T19:03:31.214237+0000","last_clean":"2023-07-14T19:03:31.214237+0000","last_became_active":"2023-07-14T19:02:30.565375+0000","last_became_peered":"2023-07-14T19:02:30.565375+0000","last_unstale":"2023-07-14T19:03:31.214237+0000","last_undegraded":"2023-07-14T19:03:31.214237+0000","last_fullsized":"2023-07-14T19:03:31.214237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:12:48.571954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210193+0000","last_change":"2023-07-14T19:02:34.577692+0000","last_active":"2023-07-14T19:03:31.210193+0000","last_peered":"2023-07-14T19:03:31.210193+0000","last_clean":"2023-07-14T19:03:31.210193+0000","last_became_active":"2023-07-14T19:02:34.577329+0000","last_became_peered":"2023-07-14T19:02:34.577329+0000","last_unstale":"2023-07-14T19:03:31.210193+0000","last_undegraded":"2023-07-14T19:03:31.210193+0000","last_fullsized":"2023-07-14T19:03:31.210193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:07.060258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212127+0000","last_change":"2023-07-14T19:02:29.938330+0000","last_active":"2023-07-14T19:03:31.212127+0000","last_peered":"2023-07-14T19:03:31.212127+0000","last_clean":"2023-07-14T19:03:31.212127+0000","last_became_active":"2023-07-14T19:02:29.938022+0000","last_became_peered":"2023-07-14T19:02:29.938022+0000","last_unstale":"2023-07-14T19:03:31.212127+0000","last_undegraded":"2023-07-14T19:03:31.212127+0000","last_fullsized":"2023-07-14T19:03:31.212127+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:24.520411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213923+0000","last_change":"2023-07-14T19:02:30.569353+0000","last_active":"2023-07-14T19:03:31.213923+0000","last_peered":"2023-07-14T19:03:31.213923+0000","last_clean":"2023-07-14T19:03:31.213923+0000","last_became_active":"2023-07-14T19:02:30.569037+0000","last_became_peered":"2023-07-14T19:02:30.569037+0000","last_unstale":"2023-07-14T19:03:31.213923+0000","last_undegraded":"2023-07-14T19:03:31.213923+0000","last_fullsized":"2023-07-14T19:03:31.213923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:38.723738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210218+0000","last_change":"2023-07-14T19:02:34.580786+0000","last_active":"2023-07-14T19:03:31.210218+0000","last_peered":"2023-07-14T19:03:31.210218+0000","last_clean":"2023-07-14T19:03:31.210218+0000","last_became_active":"2023-07-14T19:02:34.580395+0000","last_became_peered":"2023-07-14T19:02:34.580395+0000","last_unstale":"2023-07-14T19:03:31.210218+0000","last_undegraded":"2023-07-14T19:03:31.210218+0000","last_fullsized":"2023-07-14T19:03:31.210218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:45:16.010542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218850+0000","last_change":"2023-07-14T19:02:30.574302+0000","last_active":"2023-07-14T19:03:31.218850+0000","last_peered":"2023-07-14T19:03:31.218850+0000","last_clean":"2023-07-14T19:03:31.218850+0000","last_became_active":"2023-07-14T19:02:30.573912+0000","last_became_peered":"2023-07-14T19:02:30.573912+0000","last_unstale":"2023-07-14T19:03:31.218850+0000","last_undegraded":"2023-07-14T19:03:31.218850+0000","last_fullsized":"2023-07-14T19:03:31.218850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:57.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":80,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:38.687376+0000","last_change":"2023-07-14T19:02:29.942302+0000","last_active":"2023-07-14T19:03:38.687376+0000","last_peered":"2023-07-14T19:03:38.687376+0000","last_clean":"2023-07-14T19:03:38.687376+0000","last_became_active":"2023-07-14T19:02:29.942005+0000","last_became_peered":"2023-07-14T19:02:29.942005+0000","last_unstale":"2023-07-14T19:03:38.687376+0000","last_undegraded":"2023-07-14T19:03:38.687376+0000","last_fullsized":"2023-07-14T19:03:38.687376+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:32.223130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":50,"num_read_kb":50,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212188+0000","last_change":"2023-07-14T19:02:34.578248+0000","last_active":"2023-07-14T19:03:31.212188+0000","last_peered":"2023-07-14T19:03:31.212188+0000","last_clean":"2023-07-14T19:03:31.212188+0000","last_became_active":"2023-07-14T19:02:34.577930+0000","last_became_peered":"2023-07-14T19:02:34.577930+0000","last_unstale":"2023-07-14T19:03:31.212188+0000","last_undegraded":"2023-07-14T19:03:31.212188+0000","last_fullsized":"2023-07-14T19:03:31.212188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:13.074222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213462+0000","last_change":"2023-07-14T19:02:30.582892+0000","last_active":"2023-07-14T19:03:31.213462+0000","last_peered":"2023-07-14T19:03:31.213462+0000","last_clean":"2023-07-14T19:03:31.213462+0000","last_became_active":"2023-07-14T19:02:30.582588+0000","last_became_peered":"2023-07-14T19:02:30.582588+0000","last_unstale":"2023-07-14T19:03:31.213462+0000","last_undegraded":"2023-07-14T19:03:31.213462+0000","last_fullsized":"2023-07-14T19:03:31.213462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:06.056870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214203+0000","last_change":"2023-07-14T19:02:29.935857+0000","last_active":"2023-07-14T19:03:31.214203+0000","last_peered":"2023-07-14T19:03:31.214203+0000","last_clean":"2023-07-14T19:03:31.214203+0000","last_became_active":"2023-07-14T19:02:29.935489+0000","last_became_peered":"2023-07-14T19:02:29.935489+0000","last_unstale":"2023-07-14T19:03:31.214203+0000","last_undegraded":"2023-07-14T19:03:31.214203+0000","last_fullsized":"2023-07-14T19:03:31.214203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:08.956235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212941+0000","last_change":"2023-07-14T19:01:31.967476+0000","last_active":"2023-07-14T19:03:31.212941+0000","last_peered":"2023-07-14T19:03:31.212941+0000","last_clean":"2023-07-14T19:03:31.212941+0000","last_became_active":"2023-07-14T19:01:31.967146+0000","last_became_peered":"2023-07-14T19:01:31.967146+0000","last_unstale":"2023-07-14T19:03:31.212941+0000","last_undegraded":"2023-07-14T19:03:31.212941+0000","last_fullsized":"2023-07-14T19:03:31.212941+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_clean_scrub_stamp":"2023-07-14T18:58:40.634620+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:56.440561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219439+0000","last_change":"2023-07-14T19:02:34.571823+0000","last_active":"2023-07-14T19:03:31.219439+0000","last_peered":"2023-07-14T19:03:31.219439+0000","last_clean":"2023-07-14T19:03:31.219439+0000","last_became_active":"2023-07-14T19:02:34.571359+0000","last_became_peered":"2023-07-14T19:02:34.571359+0000","last_unstale":"2023-07-14T19:03:31.219439+0000","last_undegraded":"2023-07-14T19:03:31.219439+0000","last_fullsized":"2023-07-14T19:03:31.219439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:28.423706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209714+0000","last_change":"2023-07-14T19:02:30.573736+0000","last_active":"2023-07-14T19:03:31.209714+0000","last_peered":"2023-07-14T19:03:31.209714+0000","last_clean":"2023-07-14T19:03:31.209714+0000","last_became_active":"2023-07-14T19:02:30.573381+0000","last_became_peered":"2023-07-14T19:02:30.573381+0000","last_unstale":"2023-07-14T19:03:31.209714+0000","last_undegraded":"2023-07-14T19:03:31.209714+0000","last_fullsized":"2023-07-14T19:03:31.209714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:09.104343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213067+0000","last_change":"2023-07-14T19:02:29.937660+0000","last_active":"2023-07-14T19:03:31.213067+0000","last_peered":"2023-07-14T19:03:31.213067+0000","last_clean":"2023-07-14T19:03:31.213067+0000","last_became_active":"2023-07-14T19:02:29.937348+0000","last_became_peered":"2023-07-14T19:02:29.937348+0000","last_unstale":"2023-07-14T19:03:31.213067+0000","last_undegraded":"2023-07-14T19:03:31.213067+0000","last_fullsized":"2023-07-14T19:03:31.213067+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:55:06.823754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211142+0000","last_change":"2023-07-14T19:02:34.581534+0000","last_active":"2023-07-14T19:03:31.211142+0000","last_peered":"2023-07-14T19:03:31.211142+0000","last_clean":"2023-07-14T19:03:31.211142+0000","last_became_active":"2023-07-14T19:02:34.580998+0000","last_became_peered":"2023-07-14T19:02:34.580998+0000","last_unstale":"2023-07-14T19:03:31.211142+0000","last_undegraded":"2023-07-14T19:03:31.211142+0000","last_fullsized":"2023-07-14T19:03:31.211142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:05:30.391146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210585+0000","last_change":"2023-07-14T19:02:29.942336+0000","last_active":"2023-07-14T19:03:31.210585+0000","last_peered":"2023-07-14T19:03:31.210585+0000","last_clean":"2023-07-14T19:03:31.210585+0000","last_became_active":"2023-07-14T19:02:29.942033+0000","last_became_peered":"2023-07-14T19:02:29.942033+0000","last_unstale":"2023-07-14T19:03:31.210585+0000","last_undegraded":"2023-07-14T19:03:31.210585+0000","last_fullsized":"2023-07-14T19:03:31.210585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:04:44.877772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210377+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:03:31.210377+0000","last_peered":"2023-07-14T19:03:31.210377+0000","last_clean":"2023-07-14T19:03:31.210377+0000","last_became_active":"2023-07-14T19:02:30.571129+0000","last_became_peered":"2023-07-14T19:02:30.571129+0000","last_unstale":"2023-07-14T19:03:31.210377+0000","last_undegraded":"2023-07-14T19:03:31.210377+0000","last_fullsized":"2023-07-14T19:03:31.210377+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:38:56.156291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213158+0000","last_change":"2023-07-14T19:02:34.580740+0000","last_active":"2023-07-14T19:03:31.213158+0000","last_peered":"2023-07-14T19:03:31.213158+0000","last_clean":"2023-07-14T19:03:31.213158+0000","last_became_active":"2023-07-14T19:02:34.580344+0000","last_became_peered":"2023-07-14T19:02:34.580344+0000","last_unstale":"2023-07-14T19:03:31.213158+0000","last_undegraded":"2023-07-14T19:03:31.213158+0000","last_fullsized":"2023-07-14T19:03:31.213158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:33:02.155617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'5","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210352+0000","last_change":"2023-07-14T19:02:29.941349+0000","last_active":"2023-07-14T19:03:31.210352+0000","last_peered":"2023-07-14T19:03:31.210352+0000","last_clean":"2023-07-14T19:03:31.210352+0000","last_became_active":"2023-07-14T19:02:29.941039+0000","last_became_peered":"2023-07-14T19:02:29.941039+0000","last_unstale":"2023-07-14T19:03:31.210352+0000","last_undegraded":"2023-07-14T19:03:31.210352+0000","last_fullsized":"2023-07-14T19:03:31.210352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:38.367807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214004+0000","last_change":"2023-07-14T19:02:30.577546+0000","last_active":"2023-07-14T19:03:31.214004+0000","last_peered":"2023-07-14T19:03:31.214004+0000","last_clean":"2023-07-14T19:03:31.214004+0000","last_became_active":"2023-07-14T19:02:30.576833+0000","last_became_peered":"2023-07-14T19:02:30.576833+0000","last_unstale":"2023-07-14T19:03:31.214004+0000","last_undegraded":"2023-07-14T19:03:31.214004+0000","last_fullsized":"2023-07-14T19:03:31.214004+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:00:55.318416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214192+0000","last_change":"2023-07-14T19:02:34.580655+0000","last_active":"2023-07-14T19:03:31.214192+0000","last_peered":"2023-07-14T19:03:31.214192+0000","last_clean":"2023-07-14T19:03:31.214192+0000","last_became_active":"2023-07-14T19:02:34.580186+0000","last_became_peered":"2023-07-14T19:02:34.580186+0000","last_unstale":"2023-07-14T19:03:31.214192+0000","last_undegraded":"2023-07-14T19:03:31.214192+0000","last_fullsized":"2023-07-14T19:03:31.214192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:17:17.836503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212620+0000","last_change":"2023-07-14T19:02:29.938343+0000","last_active":"2023-07-14T19:03:31.212620+0000","last_peered":"2023-07-14T19:03:31.212620+0000","last_clean":"2023-07-14T19:03:31.212620+0000","last_became_active":"2023-07-14T19:02:29.938053+0000","last_became_peered":"2023-07-14T19:02:29.938053+0000","last_unstale":"2023-07-14T19:03:31.212620+0000","last_undegraded":"2023-07-14T19:03:31.212620+0000","last_fullsized":"2023-07-14T19:03:31.212620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:50:45.655040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208419+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:03:31.208419+0000","last_peered":"2023-07-14T19:03:31.208419+0000","last_clean":"2023-07-14T19:03:31.208419+0000","last_became_active":"2023-07-14T19:02:30.572436+0000","last_became_peered":"2023-07-14T19:02:30.572436+0000","last_unstale":"2023-07-14T19:03:31.208419+0000","last_undegraded":"2023-07-14T19:03:31.208419+0000","last_fullsized":"2023-07-14T19:03:31.208419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:11:10.005214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213237+0000","last_change":"2023-07-14T19:02:34.582900+0000","last_active":"2023-07-14T19:03:31.213237+0000","last_peered":"2023-07-14T19:03:31.213237+0000","last_clean":"2023-07-14T19:03:31.213237+0000","last_became_active":"2023-07-14T19:02:34.582493+0000","last_became_peered":"2023-07-14T19:02:34.582493+0000","last_unstale":"2023-07-14T19:03:31.213237+0000","last_undegraded":"2023-07-14T19:03:31.213237+0000","last_fullsized":"2023-07-14T19:03:31.213237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:00:45.030002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.280975+0000","last_change":"2023-07-14T19:02:29.941819+0000","last_active":"2023-07-14T19:03:31.280975+0000","last_peered":"2023-07-14T19:03:31.280975+0000","last_clean":"2023-07-14T19:03:31.280975+0000","last_became_active":"2023-07-14T19:02:29.941469+0000","last_became_peered":"2023-07-14T19:02:29.941469+0000","last_unstale":"2023-07-14T19:03:31.280975+0000","last_undegraded":"2023-07-14T19:03:31.280975+0000","last_fullsized":"2023-07-14T19:03:31.280975+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:28.479332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211086+0000","last_change":"2023-07-14T19:02:30.565125+0000","last_active":"2023-07-14T19:03:31.211086+0000","last_peered":"2023-07-14T19:03:31.211086+0000","last_clean":"2023-07-14T19:03:31.211086+0000","last_became_active":"2023-07-14T19:02:30.564820+0000","last_became_peered":"2023-07-14T19:02:30.564820+0000","last_unstale":"2023-07-14T19:03:31.211086+0000","last_undegraded":"2023-07-14T19:03:31.211086+0000","last_fullsized":"2023-07-14T19:03:31.211086+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:16:52.054472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.221013+0000","last_change":"2023-07-14T19:03:31.220582+0000","last_active":"2023-07-14T19:03:31.221013+0000","last_peered":"2023-07-14T19:03:31.221013+0000","last_clean":"2023-07-14T19:03:31.221013+0000","last_became_active":"2023-07-14T19:03:31.220162+0000","last_became_peered":"2023-07-14T19:03:31.220162+0000","last_unstale":"2023-07-14T19:03:31.221013+0000","last_undegraded":"2023-07-14T19:03:31.221013+0000","last_fullsized":"2023-07-14T19:03:31.221013+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:00.068361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213644+0000","last_change":"2023-07-14T19:02:29.938027+0000","last_active":"2023-07-14T19:03:31.213644+0000","last_peered":"2023-07-14T19:03:31.213644+0000","last_clean":"2023-07-14T19:03:31.213644+0000","last_became_active":"2023-07-14T19:02:29.937690+0000","last_became_peered":"2023-07-14T19:02:29.937690+0000","last_unstale":"2023-07-14T19:03:31.213644+0000","last_undegraded":"2023-07-14T19:03:31.213644+0000","last_fullsized":"2023-07-14T19:03:31.213644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:21.192935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210779+0000","last_change":"2023-07-14T19:02:30.576369+0000","last_active":"2023-07-14T19:03:31.210779+0000","last_peered":"2023-07-14T19:03:31.210779+0000","last_clean":"2023-07-14T19:03:31.210779+0000","last_became_active":"2023-07-14T19:02:30.576113+0000","last_became_peered":"2023-07-14T19:02:30.576113+0000","last_unstale":"2023-07-14T19:03:31.210779+0000","last_undegraded":"2023-07-14T19:03:31.210779+0000","last_fullsized":"2023-07-14T19:03:31.210779+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:57.280886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213432+0000","last_change":"2023-07-14T19:02:34.583724+0000","last_active":"2023-07-14T19:03:31.213432+0000","last_peered":"2023-07-14T19:03:31.213432+0000","last_clean":"2023-07-14T19:03:31.213432+0000","last_became_active":"2023-07-14T19:02:34.583218+0000","last_became_peered":"2023-07-14T19:02:34.583218+0000","last_unstale":"2023-07-14T19:03:31.213432+0000","last_undegraded":"2023-07-14T19:03:31.213432+0000","last_fullsized":"2023-07-14T19:03:31.213432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:28:38.807293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213965+0000","last_change":"2023-07-14T19:02:29.938908+0000","last_active":"2023-07-14T19:03:31.213965+0000","last_peered":"2023-07-14T19:03:31.213965+0000","last_clean":"2023-07-14T19:03:31.213965+0000","last_became_active":"2023-07-14T19:02:29.938650+0000","last_became_peered":"2023-07-14T19:02:29.938650+0000","last_unstale":"2023-07-14T19:03:31.213965+0000","last_undegraded":"2023-07-14T19:03:31.213965+0000","last_fullsized":"2023-07-14T19:03:31.213965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:22:03.037314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213600+0000","last_change":"2023-07-14T19:02:30.571870+0000","last_active":"2023-07-14T19:03:31.213600+0000","last_peered":"2023-07-14T19:03:31.213600+0000","last_clean":"2023-07-14T19:03:31.213600+0000","last_became_active":"2023-07-14T19:02:30.571521+0000","last_became_peered":"2023-07-14T19:02:30.571521+0000","last_unstale":"2023-07-14T19:03:31.213600+0000","last_undegraded":"2023-07-14T19:03:31.213600+0000","last_fullsized":"2023-07-14T19:03:31.213600+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:47.125420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213270+0000","last_change":"2023-07-14T19:02:34.589900+0000","last_active":"2023-07-14T19:03:31.213270+0000","last_peered":"2023-07-14T19:03:31.213270+0000","last_clean":"2023-07-14T19:03:31.213270+0000","last_became_active":"2023-07-14T19:02:34.589481+0000","last_became_peered":"2023-07-14T19:02:34.589481+0000","last_unstale":"2023-07-14T19:03:31.213270+0000","last_undegraded":"2023-07-14T19:03:31.213270+0000","last_fullsized":"2023-07-14T19:03:31.213270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:07.853098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212669+0000","last_change":"2023-07-14T19:02:29.940315+0000","last_active":"2023-07-14T19:03:31.212669+0000","last_peered":"2023-07-14T19:03:31.212669+0000","last_clean":"2023-07-14T19:03:31.212669+0000","last_became_active":"2023-07-14T19:02:29.939991+0000","last_became_peered":"2023-07-14T19:02:29.939991+0000","last_unstale":"2023-07-14T19:03:31.212669+0000","last_undegraded":"2023-07-14T19:03:31.212669+0000","last_fullsized":"2023-07-14T19:03:31.212669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:30.521480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212329+0000","last_change":"2023-07-14T19:02:30.583959+0000","last_active":"2023-07-14T19:03:31.212329+0000","last_peered":"2023-07-14T19:03:31.212329+0000","last_clean":"2023-07-14T19:03:31.212329+0000","last_became_active":"2023-07-14T19:02:30.583657+0000","last_became_peered":"2023-07-14T19:02:30.583657+0000","last_unstale":"2023-07-14T19:03:31.212329+0000","last_undegraded":"2023-07-14T19:03:31.212329+0000","last_fullsized":"2023-07-14T19:03:31.212329+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:04:07.282087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281103+0000","last_change":"2023-07-14T19:02:34.579328+0000","last_active":"2023-07-14T19:03:31.281103+0000","last_peered":"2023-07-14T19:03:31.281103+0000","last_clean":"2023-07-14T19:03:31.281103+0000","last_became_active":"2023-07-14T19:02:34.578904+0000","last_became_peered":"2023-07-14T19:02:34.578904+0000","last_unstale":"2023-07-14T19:03:31.281103+0000","last_undegraded":"2023-07-14T19:03:31.281103+0000","last_fullsized":"2023-07-14T19:03:31.281103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:15.218624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214589+0000","last_change":"2023-07-14T19:02:29.940972+0000","last_active":"2023-07-14T19:03:31.214589+0000","last_peered":"2023-07-14T19:03:31.214589+0000","last_clean":"2023-07-14T19:03:31.214589+0000","last_became_active":"2023-07-14T19:02:29.940670+0000","last_became_peered":"2023-07-14T19:02:29.940670+0000","last_unstale":"2023-07-14T19:03:31.214589+0000","last_undegraded":"2023-07-14T19:03:31.214589+0000","last_fullsized":"2023-07-14T19:03:31.214589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:55:01.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212697+0000","last_change":"2023-07-14T19:02:30.583714+0000","last_active":"2023-07-14T19:03:31.212697+0000","last_peered":"2023-07-14T19:03:31.212697+0000","last_clean":"2023-07-14T19:03:31.212697+0000","last_became_active":"2023-07-14T19:02:30.583404+0000","last_became_peered":"2023-07-14T19:02:30.583404+0000","last_unstale":"2023-07-14T19:03:31.212697+0000","last_undegraded":"2023-07-14T19:03:31.212697+0000","last_fullsized":"2023-07-14T19:03:31.212697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:55.354431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207943+0000","last_change":"2023-07-14T19:02:34.582985+0000","last_active":"2023-07-14T19:03:31.207943+0000","last_peered":"2023-07-14T19:03:31.207943+0000","last_clean":"2023-07-14T19:03:31.207943+0000","last_became_active":"2023-07-14T19:02:34.582606+0000","last_became_peered":"2023-07-14T19:02:34.582606+0000","last_unstale":"2023-07-14T19:03:31.207943+0000","last_undegraded":"2023-07-14T19:03:31.207943+0000","last_fullsized":"2023-07-14T19:03:31.207943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:55:50.958196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219531+0000","last_change":"2023-07-14T19:02:29.944064+0000","last_active":"2023-07-14T19:03:31.219531+0000","last_peered":"2023-07-14T19:03:31.219531+0000","last_clean":"2023-07-14T19:03:31.219531+0000","last_became_active":"2023-07-14T19:02:29.943770+0000","last_became_peered":"2023-07-14T19:02:29.943770+0000","last_unstale":"2023-07-14T19:03:31.219531+0000","last_undegraded":"2023-07-14T19:03:31.219531+0000","last_fullsized":"2023-07-14T19:03:31.219531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:02:39.670163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213041+0000","last_change":"2023-07-14T19:02:30.580668+0000","last_active":"2023-07-14T19:03:31.213041+0000","last_peered":"2023-07-14T19:03:31.213041+0000","last_clean":"2023-07-14T19:03:31.213041+0000","last_became_active":"2023-07-14T19:02:30.580297+0000","last_became_peered":"2023-07-14T19:02:30.580297+0000","last_unstale":"2023-07-14T19:03:31.213041+0000","last_undegraded":"2023-07-14T19:03:31.213041+0000","last_fullsized":"2023-07-14T19:03:31.213041+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:03:37.240639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.217879+0000","last_change":"2023-07-14T19:02:34.578894+0000","last_active":"2023-07-14T19:03:31.217879+0000","last_peered":"2023-07-14T19:03:31.217879+0000","last_clean":"2023-07-14T19:03:31.217879+0000","last_became_active":"2023-07-14T19:02:34.578463+0000","last_became_peered":"2023-07-14T19:02:34.578463+0000","last_unstale":"2023-07-14T19:03:31.217879+0000","last_undegraded":"2023-07-14T19:03:31.217879+0000","last_fullsized":"2023-07-14T19:03:31.217879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:53.046945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210215+0000","last_change":"2023-07-14T19:02:29.938430+0000","last_active":"2023-07-14T19:03:31.210215+0000","last_peered":"2023-07-14T19:03:31.210215+0000","last_clean":"2023-07-14T19:03:31.210215+0000","last_became_active":"2023-07-14T19:02:29.938118+0000","last_became_peered":"2023-07-14T19:02:29.938118+0000","last_unstale":"2023-07-14T19:03:31.210215+0000","last_undegraded":"2023-07-14T19:03:31.210215+0000","last_fullsized":"2023-07-14T19:03:31.210215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:59.720559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212504+0000","last_change":"2023-07-14T19:02:30.575124+0000","last_active":"2023-07-14T19:03:31.212504+0000","last_peered":"2023-07-14T19:03:31.212504+0000","last_clean":"2023-07-14T19:03:31.212504+0000","last_became_active":"2023-07-14T19:02:30.574805+0000","last_became_peered":"2023-07-14T19:02:30.574805+0000","last_unstale":"2023-07-14T19:03:31.212504+0000","last_undegraded":"2023-07-14T19:03:31.212504+0000","last_fullsized":"2023-07-14T19:03:31.212504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:42:48.292682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210923+0000","last_change":"2023-07-14T19:02:30.573897+0000","last_active":"2023-07-14T19:03:31.210923+0000","last_peered":"2023-07-14T19:03:31.210923+0000","last_clean":"2023-07-14T19:03:31.210923+0000","last_became_active":"2023-07-14T19:02:30.573585+0000","last_became_peered":"2023-07-14T19:02:30.573585+0000","last_unstale":"2023-07-14T19:03:31.210923+0000","last_undegraded":"2023-07-14T19:03:31.210923+0000","last_fullsized":"2023-07-14T19:03:31.210923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:26:13.564561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215202+0000","last_change":"2023-07-14T19:02:30.572390+0000","last_active":"2023-07-14T19:03:31.215202+0000","last_peered":"2023-07-14T19:03:31.215202+0000","last_clean":"2023-07-14T19:03:31.215202+0000","last_became_active":"2023-07-14T19:02:30.572093+0000","last_became_peered":"2023-07-14T19:02:30.572093+0000","last_unstale":"2023-07-14T19:03:31.215202+0000","last_undegraded":"2023-07-14T19:03:31.215202+0000","last_fullsized":"2023-07-14T19:03:31.215202+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:57:10.086878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213816+0000","last_change":"2023-07-14T19:02:30.572548+0000","last_active":"2023-07-14T19:03:31.213816+0000","last_peered":"2023-07-14T19:03:31.213816+0000","last_clean":"2023-07-14T19:03:31.213816+0000","last_became_active":"2023-07-14T19:02:30.572260+0000","last_became_peered":"2023-07-14T19:02:30.572260+0000","last_unstale":"2023-07-14T19:03:31.213816+0000","last_undegraded":"2023-07-14T19:03:31.213816+0000","last_fullsized":"2023-07-14T19:03:31.213816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:55.025634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208138+0000","last_change":"2023-07-14T19:02:30.569856+0000","last_active":"2023-07-14T19:03:31.208138+0000","last_peered":"2023-07-14T19:03:31.208138+0000","last_clean":"2023-07-14T19:03:31.208138+0000","last_became_active":"2023-07-14T19:02:30.569546+0000","last_became_peered":"2023-07-14T19:02:30.569546+0000","last_unstale":"2023-07-14T19:03:31.208138+0000","last_undegraded":"2023-07-14T19:03:31.208138+0000","last_fullsized":"2023-07-14T19:03:31.208138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:43.152189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210350+0000","last_change":"2023-07-14T19:02:30.570836+0000","last_active":"2023-07-14T19:03:31.210350+0000","last_peered":"2023-07-14T19:03:31.210350+0000","last_clean":"2023-07-14T19:03:31.210350+0000","last_became_active":"2023-07-14T19:02:30.569675+0000","last_became_peered":"2023-07-14T19:02:30.569675+0000","last_unstale":"2023-07-14T19:03:31.210350+0000","last_undegraded":"2023-07-14T19:03:31.210350+0000","last_fullsized":"2023-07-14T19:03:31.210350+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:26:19.766316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209916+0000","last_change":"2023-07-14T19:02:30.566846+0000","last_active":"2023-07-14T19:03:31.209916+0000","last_peered":"2023-07-14T19:03:31.209916+0000","last_clean":"2023-07-14T19:03:31.209916+0000","last_became_active":"2023-07-14T19:02:30.566525+0000","last_became_peered":"2023-07-14T19:02:30.566525+0000","last_unstale":"2023-07-14T19:03:31.209916+0000","last_undegraded":"2023-07-14T19:03:31.209916+0000","last_fullsized":"2023-07-14T19:03:31.209916+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:12:37.744089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215127+0000","last_change":"2023-07-14T19:02:30.579680+0000","last_active":"2023-07-14T19:03:31.215127+0000","last_peered":"2023-07-14T19:03:31.215127+0000","last_clean":"2023-07-14T19:03:31.215127+0000","last_became_active":"2023-07-14T19:02:30.579379+0000","last_became_peered":"2023-07-14T19:02:30.579379+0000","last_unstale":"2023-07-14T19:03:31.215127+0000","last_undegraded":"2023-07-14T19:03:31.215127+0000","last_fullsized":"2023-07-14T19:03:31.215127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:00:25.682516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214145+0000","last_change":"2023-07-14T19:02:30.577722+0000","last_active":"2023-07-14T19:03:31.214145+0000","last_peered":"2023-07-14T19:03:31.214145+0000","last_clean":"2023-07-14T19:03:31.214145+0000","last_became_active":"2023-07-14T19:02:30.577247+0000","last_became_peered":"2023-07-14T19:02:30.577247+0000","last_unstale":"2023-07-14T19:03:31.214145+0000","last_undegraded":"2023-07-14T19:03:31.214145+0000","last_fullsized":"2023-07-14T19:03:31.214145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:19:14.172473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218436+0000","last_change":"2023-07-14T19:02:30.571052+0000","last_active":"2023-07-14T19:03:31.218436+0000","last_peered":"2023-07-14T19:03:31.218436+0000","last_clean":"2023-07-14T19:03:31.218436+0000","last_became_active":"2023-07-14T19:02:30.570722+0000","last_became_peered":"2023-07-14T19:02:30.570722+0000","last_unstale":"2023-07-14T19:03:31.218436+0000","last_undegraded":"2023-07-14T19:03:31.218436+0000","last_fullsized":"2023-07-14T19:03:31.218436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:50:37.718571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213772+0000","last_change":"2023-07-14T19:02:30.575685+0000","last_active":"2023-07-14T19:03:31.213772+0000","last_peered":"2023-07-14T19:03:31.213772+0000","last_clean":"2023-07-14T19:03:31.213772+0000","last_became_active":"2023-07-14T19:02:30.575379+0000","last_became_peered":"2023-07-14T19:02:30.575379+0000","last_unstale":"2023-07-14T19:03:31.213772+0000","last_undegraded":"2023-07-14T19:03:31.213772+0000","last_fullsized":"2023-07-14T19:03:31.213772+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:14:01.199281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281456+0000","last_change":"2023-07-14T19:02:30.582450+0000","last_active":"2023-07-14T19:03:31.281456+0000","last_peered":"2023-07-14T19:03:31.281456+0000","last_clean":"2023-07-14T19:03:31.281456+0000","last_became_active":"2023-07-14T19:02:30.582074+0000","last_became_peered":"2023-07-14T19:02:30.582074+0000","last_unstale":"2023-07-14T19:03:31.281456+0000","last_undegraded":"2023-07-14T19:03:31.281456+0000","last_fullsized":"2023-07-14T19:03:31.281456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:50.575652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210265+0000","last_change":"2023-07-14T19:02:30.573042+0000","last_active":"2023-07-14T19:03:31.210265+0000","last_peered":"2023-07-14T19:03:31.210265+0000","last_clean":"2023-07-14T19:03:31.210265+0000","last_became_active":"2023-07-14T19:02:30.572498+0000","last_became_peered":"2023-07-14T19:02:30.572498+0000","last_unstale":"2023-07-14T19:03:31.210265+0000","last_undegraded":"2023-07-14T19:03:31.210265+0000","last_fullsized":"2023-07-14T19:03:31.210265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:01.947296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210278+0000","last_change":"2023-07-14T19:02:30.569441+0000","last_active":"2023-07-14T19:03:31.210278+0000","last_peered":"2023-07-14T19:03:31.210278+0000","last_clean":"2023-07-14T19:03:31.210278+0000","last_became_active":"2023-07-14T19:02:30.569180+0000","last_became_peered":"2023-07-14T19:02:30.569180+0000","last_unstale":"2023-07-14T19:03:31.210278+0000","last_undegraded":"2023-07-14T19:03:31.210278+0000","last_fullsized":"2023-07-14T19:03:31.210278+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:24.866934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61264,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":324,"num_read_kb":224,"num_write":153,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":146,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514522,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":476824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.64700000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.57199999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.26200000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61199999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.71299999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.34599999999999997}]},{"osd":6,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.59799999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.52200000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.38400000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.69599999999999995},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.58599999999999997}]},{"osd":10,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.74299999999999999}]}]},{"osd":10,"up_from":69,"seq":296352743453,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1654784,"data_stored":1121258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.62}]},{"osd":1,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.26900000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.63100000000000001,"15min":0.621},"min":{"1min":0.48499999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.69599999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.442,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45400000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79000000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.68400000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59999999999999998,"15min":0.60099999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.316},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.374}]},{"osd":5,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.47099999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.56399999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.55200000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69499999999999995,"15min":0.69299999999999995},"min":{"1min":0.51200000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89100000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.59299999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77800000000000002,"15min":0.77900000000000003},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96699999999999997,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.56399999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77800000000000002,"15min":0.78200000000000003},"min":{"1min":0.59599999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.747,"15min":0.753},"min":{"1min":0.57199999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88,"5min":1.226,"15min":1.226},"last":0.70699999999999996}]},{"osd":11,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.63200000000000001}]}]},{"osd":9,"up_from":64,"seq":274877906977,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":481010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.42099999999999999,"5min":0.63,"15min":0.63},"last":0.37},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.33400000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79000000000000004,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.52700000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.71299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.432}]},{"osd":3,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47699999999999998,"15min":0.49299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66300000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.497},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.35299999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.41099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72499999999999998,"5min":1.008,"15min":1.008},"last":0.627},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.746,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56100000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70499999999999996,"5min":1.026,"15min":1.026},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.45900000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.62},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.69999999999999996,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.81799999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.63700000000000001}]},{"osd":8,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.73699999999999999,"15min":0.755},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81399999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84299999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.53800000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.746,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.746},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.70399999999999996,"15min":0.71499999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79400000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.78200000000000003}]},{"osd":11,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.55000000000000004}]}]},{"osd":8,"up_from":57,"seq":244813135910,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":476391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88300000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.62},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47799999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90900000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76300000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.53300000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.498},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.42799999999999999}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.878,"15min":0.878},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.70099999999999996}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.746},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.68400000000000005,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.73399999999999999}]},{"osd":5,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.626,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.77100000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.94299999999999995,"5min":1.01,"15min":1.01},"last":0.78400000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.871,"15min":0.871},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0369999999999999,"5min":1.046,"15min":1.046},"last":0.84399999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.755,"15min":0.75},"min":{"1min":0.53200000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.71299999999999997}]},{"osd":10,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77600000000000002,"15min":0.76600000000000001},"min":{"1min":0.64600000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.016,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.81799999999999995},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.70299999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0700000000000001,"5min":1.081,"15min":1.081},"last":0.85799999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.871},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.82999999999999996}]}]},{"osd":7,"up_from":50,"seq":214748364843,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1576960,"data_stored":1072763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56999999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.40899999999999997},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.621}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57399999999999995,"15min":0.54900000000000004},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.70199999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71699999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.68600000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.47699999999999998,"15min":0.45900000000000002},"min":{"1min":0.38300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.48599999999999999},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69999999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.65100000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.627,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.87,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.75600000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.63},"min":{"1min":0.54100000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.747},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94899999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.82299999999999995}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.89100000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74099999999999999,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96999999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.72799999999999998},{"interface":"front","average":{"1min":0.997,"5min":0.76300000000000001,"15min":0.72399999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":6.8970000000000002,"5min":6.8970000000000002,"15min":6.8970000000000002},"last":0.871}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.72299999999999998},"min":{"1min":0.60599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95299999999999996,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":0.745,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.64700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.92200000000000004}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.95399999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.90600000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528367,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":472090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39900000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.54700000000000004,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41999999999999998,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56200000000000006,"5min":1.591,"15min":1.591},"last":0.35499999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.55700000000000005},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83699999999999997,"5min":1.159,"15min":1.159},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":1.49,"15min":1.49},"last":0.39200000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73799999999999999,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.59899999999999998},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61099999999999999,"15min":0.59299999999999997},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.77900000000000003,"5min":1.419,"15min":1.419},"last":0.58599999999999997}]},{"osd":3,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.34300000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.48199999999999998,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":2.077,"15min":2.077},"last":0.32400000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85299999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.497},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64600000000000002,"15min":0.624},"min":{"1min":0.45500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80900000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.41199999999999998}]},{"osd":5,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65100000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76200000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.66600000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.66800000000000004,"15min":0.67900000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76300000000000001,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.75,"15min":0.73099999999999998},"min":{"1min":0.51500000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.86699999999999999,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.61299999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73599999999999999},"min":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.629}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.77400000000000002,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78900000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.441}]},{"osd":10,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82499999999999996,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.55700000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.88600000000000001,"5min":1.234,"15min":1.234},"last":0.65100000000000002}]},{"osd":11,"last update":"Fri Jul 14 19:03:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.68999999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":528970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52400000000000002,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.3370000000000002},"last":0.434},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.379,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.55200000000000005,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.52500000000000002,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.66500000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88300000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81999999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.53300000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.44600000000000001,"15min":0.39100000000000001},"min":{"1min":0.36599999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44600000000000001,"15min":0.40200000000000002},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.46100000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.59999999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61799999999999999,"15min":0.59199999999999997},"min":{"1min":0.53800000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.65000000000000002,"15min":0.60199999999999998},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.3209999999999997,"5min":4.3209999999999997,"15min":4.3209999999999997},"last":0.63300000000000001}]},{"osd":7,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.54900000000000004,"15min":0.51700000000000002},"min":{"1min":0.439,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.628,"15min":0.58199999999999996},"min":{"1min":0.52200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.751}]},{"osd":8,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72899999999999998,"15min":0.71599999999999997},"min":{"1min":0.498,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.99299999999999999,"5min":0.996,"15min":0.996},"last":0.81299999999999994}]},{"osd":9,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.79000000000000004,"15min":0.77300000000000002},"min":{"1min":0.54800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.60099999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.85799999999999998}]},{"osd":10,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.77000000000000002,"15min":0.76000000000000001},"min":{"1min":0.625,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.73199999999999998,"15min":0.72499999999999998},"min":{"1min":0.60699999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.90300000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.73499999999999999}]},{"osd":11,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.877},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.77000000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":473205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.159,"15min":0.159},"max":{"1min":0.72499999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.48999999999999999},"min":{"1min":0.40300000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.75800000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.65200000000000002}]},{"osd":2,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.37,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.746,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.78900000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.505},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.55700000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.372,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68700000000000006,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.50900000000000001},"min":{"1min":0.43099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.77900000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.46000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91700000000000004,"5min":1.042,"15min":1.042},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.60899999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.496,"15min":0.495},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72099999999999997,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.59699999999999998}]},{"osd":7,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.76700000000000002},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.621}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70199999999999996,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.024,"15min":1.024},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.63500000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.78200000000000003,"5min":1.204,"15min":1.204},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72499999999999998,"15min":0.746},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82199999999999995,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.52200000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.877,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.754},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.79700000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.72899999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.71899999999999997}]}]},{"osd":2,"up_from":18,"seq":77309411392,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":530628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78700000000000003,"5min":1.111,"15min":1.111},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.46800000000000003}]},{"osd":1,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.628,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.73299999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77000000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55200000000000005,"15min":0.53900000000000003},"min":{"1min":0.45500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78100000000000003,"5min":1.046,"15min":1.046},"last":0.76400000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.746}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.44500000000000001,"15min":0.41799999999999998},"min":{"1min":0.33600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.435},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.59699999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.59599999999999997,"15min":0.56599999999999995},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60299999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89300000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.69599999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.92100000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.443,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.59599999999999997,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.71199999999999997}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92600000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72799999999999998,"15min":0.71999999999999997},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.79100000000000004}]},{"osd":10,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.753,"15min":0.73099999999999998},"min":{"1min":0.69799999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.752,"15min":0.72699999999999998},"min":{"1min":0.68700000000000006,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":0.88100000000000001}]},{"osd":11,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":0.91200000000000003}]}]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":472284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.188,"15min":0.188},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35299999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.215,"15min":0.215},"max":{"1min":0.52800000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.28299999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71799999999999997,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.53600000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.39600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.745,"5min":4.6509999999999998,"15min":4.6509999999999998},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60799999999999998,"15min":0.66400000000000003},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.47199999999999998}]},{"osd":4,"last update":"Fri Jul 14 19:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.498},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.621}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.80800000000000005},{"interface":"front","average":{"1min":0.621,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.498,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.81699999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.82799999999999996,"5min":1.119,"15min":1.119},"last":0.36499999999999999}]},{"osd":7,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.376,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.53000000000000003}]},{"osd":8,"last update":"Fri Jul 14 19:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66200000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76000000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.39600000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.68899999999999995,"15min":0.69699999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91900000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69799999999999995,"15min":0.70399999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.59799999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.79600000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.69699999999999995}]}]},{"osd":4,"up_from":33,"seq":141733920824,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":476809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.503,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.38100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38400000000000001,"15min":0.34799999999999998},"min":{"1min":0.33600000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.59199999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40100000000000002,"15min":0.36499999999999999},"min":{"1min":0.34399999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.71799999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.89500000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.58499999999999996,"15min":0.54300000000000004},"min":{"1min":0.45500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.105,"15min":1.105},"last":0.82099999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55300000000000005,"15min":0.51200000000000001},"min":{"1min":0.432,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72299999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54900000000000004,"15min":0.50600000000000001},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.58699999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57599999999999996,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.72399999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.60399999999999998},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":1.038,"15min":1.038},"last":0.64500000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.442,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.873,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.86099999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68600000000000005,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.90500000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.746},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.58399999999999996,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.874}]},{"osd":10,"last update":"Fri Jul 14 19:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68700000000000006,"15min":0.68500000000000005},"min":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71799999999999997,"15min":0.71199999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.93300000000000005}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.96999999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757299,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1067951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57299999999999995}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.69299999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.41399999999999998,"15min":0.38900000000000001},"min":{"1min":0.309,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70599999999999996,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.63100000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60199999999999998,"15min":0.58799999999999997},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84099999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.751},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64500000000000002,"15min":0.63100000000000001},"min":{"1min":0.40200000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.78700000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63900000000000001,"15min":0.622},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.65100000000000002,"15min":0.64000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.80800000000000005}]},{"osd":8,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77600000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.753},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.86899999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.71399999999999997,"15min":0.69699999999999995},"min":{"1min":0.67500000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.71099999999999997,"15min":0.69799999999999995},"min":{"1min":0.623,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.85399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:03:43.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:43 smithi078 ceph-mon[127425]: pgmap v293: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:43.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:43 smithi102 ceph-mon[129280]: pgmap v293: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:43.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:43 smithi088 ceph-mon[128973]: pgmap v293: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 6 op/s; 13 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:44.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:44 smithi078 ceph-mon[127425]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:44.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:44 smithi102 ceph-mon[129280]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:44.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:44 smithi088 ceph-mon[128973]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:45.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:45 smithi078 ceph-mon[127425]: pgmap v294: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:45.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:45 smithi102 ceph-mon[129280]: pgmap v294: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:45.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:45 smithi088 ceph-mon[128973]: pgmap v294: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:46.329 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph pg dump --format=json 2023-07-14T19:03:46.896 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:46.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:46 smithi088 ceph-mon[128973]: pgmap v295: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:47.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:46 smithi078 ceph-mon[127425]: pgmap v295: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:47.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:46 smithi102 ceph-mon[129280]: pgmap v295: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 12 B/s, 1 keys/s, 1 objects/s recovering 2023-07-14T19:03:48.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 -- 172.21.15.78:0/725160116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 msgr2=0x7ff80810cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/725160116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 0x7ff80810cb30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff80000b0a0 tx=0x7ff80000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 -- 172.21.15.78:0/725160116 shutdown_connections 2023-07-14T19:03:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/725160116 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 0x7ff80810cb30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/725160116 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff808104c70 0x7ff808105040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/725160116 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8080fe210 0x7ff8080fe680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 -- 172.21.15.78:0/725160116 >> 172.21.15.78:0/725160116 conn(0x7ff808075280 msgr2=0x7ff808075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:48.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 -- 172.21.15.78:0/725160116 shutdown_connections 2023-07-14T19:03:48.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.213+0000 7ff80ceaa700 1 -- 172.21.15.78:0/725160116 wait complete. 2023-07-14T19:03:48.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff80ceaa700 1 Processor -- start 2023-07-14T19:03:48.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff80ceaa700 1 -- start start 2023-07-14T19:03:48.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff80ceaa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8080fe210 0x7ff808195fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff80ceaa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 0x7ff8081964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff80ceaa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff808104c70 0x7ff80819a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff80ceaa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80819af60 con 0x7ff8080febc0 2023-07-14T19:03:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff80ceaa700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80819b0d0 con 0x7ff8080fe210 2023-07-14T19:03:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff80ceaa700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff80819b290 con 0x7ff808104c70 2023-07-14T19:03:48.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.214+0000 7ff805d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 0x7ff8081964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:48.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff805d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 0x7ff8081964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35508/0 (socket says 172.21.15.78:35508) 2023-07-14T19:03:48.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff805d9b700 1 -- 172.21.15.78:0/1105295152 learned_addr learned my addr 172.21.15.78:0/1105295152 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:48.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff806d9d700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff808104c70 0x7ff80819a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:48.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff80659c700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8080fe210 0x7ff808195fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:48.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff805d9b700 1 -- 172.21.15.78:0/1105295152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff808104c70 msgr2=0x7ff80819a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:48.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff805d9b700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff808104c70 0x7ff80819a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff805d9b700 1 -- 172.21.15.78:0/1105295152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8080fe210 msgr2=0x7ff808195fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:48.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff805d9b700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8080fe210 0x7ff808195fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff805d9b700 1 -- 172.21.15.78:0/1105295152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff800009d00 con 0x7ff8080febc0 2023-07-14T19:03:48.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff805d9b700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 0x7ff8081964e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800b530 tx=0x7ff7f800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:48.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff7f77fe700 1 -- 172.21.15.78:0/1105295152 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f8011830 con 0x7ff8080febc0 2023-07-14T19:03:48.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff7f77fe700 1 -- 172.21.15.78:0/1105295152 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff7f8011e20 con 0x7ff8080febc0 2023-07-14T19:03:48.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80819b570 con 0x7ff8080febc0 2023-07-14T19:03:48.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff7f77fe700 1 -- 172.21.15.78:0/1105295152 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7f800f2e0 con 0x7ff8080febc0 2023-07-14T19:03:48.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.215+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8081a2dd0 con 0x7ff8080febc0 2023-07-14T19:03:48.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.216+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff80804e340 con 0x7ff8080febc0 2023-07-14T19:03:48.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.217+0000 7ff7f77fe700 1 -- 172.21.15.78:0/1105295152 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff7f8011990 con 0x7ff8080febc0 2023-07-14T19:03:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.218+0000 7ff7f77fe700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff7f00a1260 0x7ff7f00a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.218+0000 7ff80659c700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff7f00a1260 0x7ff7f00a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.218+0000 7ff7f77fe700 1 -- 172.21.15.78:0/1105295152 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7ff7f80ccbb0 con 0x7ff8080febc0 2023-07-14T19:03:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.218+0000 7ff80659c700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff7f00a1260 0x7ff7f00a3710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc006fd0 tx=0x7ff7fc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:48.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.221+0000 7ff7f77fe700 1 -- 172.21.15.78:0/1105295152 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff7f8095f40 con 0x7ff8080febc0 2023-07-14T19:03:48.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.404+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff808197300 con 0x7ff7f00a1260 2023-07-14T19:03:48.419 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.415+0000 7ff7f77fe700 1 -- 172.21.15.78:0/1105295152 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+427286 (secure 0 0 0) 0x7ff808197300 con 0x7ff7f00a1260 2023-07-14T19:03:48.423 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff7f00a1260 msgr2=0x7ff7f00a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff7f00a1260 0x7ff7f00a3710 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ff7fc006fd0 tx=0x7ff7fc008040 comp rx=0 tx=0).stop 2023-07-14T19:03:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 msgr2=0x7ff8081964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:48.425 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 0x7ff8081964e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff7f800b530 tx=0x7ff7f800b840 comp rx=0 tx=0).stop 2023-07-14T19:03:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 shutdown_connections 2023-07-14T19:03:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff7f00a1260 0x7ff7f00a3710 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff8080fe210 0x7ff808195fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff808104c70 0x7ff80819a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 --2- 172.21.15.78:0/1105295152 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff8080febc0 0x7ff8081964e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 >> 172.21.15.78:0/1105295152 conn(0x7ff808075280 msgr2=0x7ff8080fbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:48.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 shutdown_connections 2023-07-14T19:03:48.427 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:48.419+0000 7ff80ceaa700 1 -- 172.21.15.78:0/1105295152 wait complete. 2023-07-14T19:03:48.427 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2023-07-14T19:03:48.845 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":296,"stamp":"2023-07-14T19:03:48.129818+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":651878,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":480,"num_read_kb":348,"num_write":336,"num_write_kb":3047,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":1180882,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":274,"ondisk_log_size":274,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":337696,"kb_used_data":13428,"kb_used_omap":20,"kb_used_meta":324075,"kb_avail":1124604128,"statfs":{"total":1151940427776,"available":1151594627072,"internally_reserved":0,"allocated":13750272,"data_stored":7650183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":21372,"internal_metadata":331852932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":70,"num_read_kb":30,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003587"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210016+0000","last_change":"2023-07-14T19:02:30.582427+0000","last_active":"2023-07-14T19:03:31.210016+0000","last_peered":"2023-07-14T19:03:31.210016+0000","last_clean":"2023-07-14T19:03:31.210016+0000","last_became_active":"2023-07-14T19:02:30.582149+0000","last_became_peered":"2023-07-14T19:02:30.582149+0000","last_unstale":"2023-07-14T19:03:31.210016+0000","last_undegraded":"2023-07-14T19:03:31.210016+0000","last_fullsized":"2023-07-14T19:03:31.210016+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:01:09.726556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208324+0000","last_change":"2023-07-14T19:02:30.567053+0000","last_active":"2023-07-14T19:03:31.208324+0000","last_peered":"2023-07-14T19:03:31.208324+0000","last_clean":"2023-07-14T19:03:31.208324+0000","last_became_active":"2023-07-14T19:02:30.566736+0000","last_became_peered":"2023-07-14T19:02:30.566736+0000","last_unstale":"2023-07-14T19:03:31.208324+0000","last_undegraded":"2023-07-14T19:03:31.208324+0000","last_fullsized":"2023-07-14T19:03:31.208324+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:44:23.099904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208244+0000","last_change":"2023-07-14T19:02:30.577632+0000","last_active":"2023-07-14T19:03:31.208244+0000","last_peered":"2023-07-14T19:03:31.208244+0000","last_clean":"2023-07-14T19:03:31.208244+0000","last_became_active":"2023-07-14T19:02:30.577300+0000","last_became_peered":"2023-07-14T19:02:30.577300+0000","last_unstale":"2023-07-14T19:03:31.208244+0000","last_undegraded":"2023-07-14T19:03:31.208244+0000","last_fullsized":"2023-07-14T19:03:31.208244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:15:18.165242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213866+0000","last_change":"2023-07-14T19:02:30.581405+0000","last_active":"2023-07-14T19:03:31.213866+0000","last_peered":"2023-07-14T19:03:31.213866+0000","last_clean":"2023-07-14T19:03:31.213866+0000","last_became_active":"2023-07-14T19:02:30.581113+0000","last_became_peered":"2023-07-14T19:02:30.581113+0000","last_unstale":"2023-07-14T19:03:31.213866+0000","last_undegraded":"2023-07-14T19:03:31.213866+0000","last_fullsized":"2023-07-14T19:03:31.213866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:42:53.074489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212421+0000","last_change":"2023-07-14T19:02:30.574285+0000","last_active":"2023-07-14T19:03:31.212421+0000","last_peered":"2023-07-14T19:03:31.212421+0000","last_clean":"2023-07-14T19:03:31.212421+0000","last_became_active":"2023-07-14T19:02:30.573967+0000","last_became_peered":"2023-07-14T19:02:30.573967+0000","last_unstale":"2023-07-14T19:03:31.212421+0000","last_undegraded":"2023-07-14T19:03:31.212421+0000","last_fullsized":"2023-07-14T19:03:31.212421+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:39:39.079558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215131+0000","last_change":"2023-07-14T19:02:30.574135+0000","last_active":"2023-07-14T19:03:31.215131+0000","last_peered":"2023-07-14T19:03:31.215131+0000","last_clean":"2023-07-14T19:03:31.215131+0000","last_became_active":"2023-07-14T19:02:30.573821+0000","last_became_peered":"2023-07-14T19:02:30.573821+0000","last_unstale":"2023-07-14T19:03:31.215131+0000","last_undegraded":"2023-07-14T19:03:31.215131+0000","last_fullsized":"2023-07-14T19:03:31.215131+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:05:56.064229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212782+0000","last_change":"2023-07-14T19:02:30.578051+0000","last_active":"2023-07-14T19:03:31.212782+0000","last_peered":"2023-07-14T19:03:31.212782+0000","last_clean":"2023-07-14T19:03:31.212782+0000","last_became_active":"2023-07-14T19:02:30.577700+0000","last_became_peered":"2023-07-14T19:02:30.577700+0000","last_unstale":"2023-07-14T19:03:31.212782+0000","last_undegraded":"2023-07-14T19:03:31.212782+0000","last_fullsized":"2023-07-14T19:03:31.212782+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:40:01.289258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213306+0000","last_change":"2023-07-14T19:02:30.567480+0000","last_active":"2023-07-14T19:03:31.213306+0000","last_peered":"2023-07-14T19:03:31.213306+0000","last_clean":"2023-07-14T19:03:31.213306+0000","last_became_active":"2023-07-14T19:02:30.567147+0000","last_became_peered":"2023-07-14T19:02:30.567147+0000","last_unstale":"2023-07-14T19:03:31.213306+0000","last_undegraded":"2023-07-14T19:03:31.213306+0000","last_fullsized":"2023-07-14T19:03:31.213306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:02:57.085021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281462+0000","last_change":"2023-07-14T19:02:30.581936+0000","last_active":"2023-07-14T19:03:31.281462+0000","last_peered":"2023-07-14T19:03:31.281462+0000","last_clean":"2023-07-14T19:03:31.281462+0000","last_became_active":"2023-07-14T19:02:30.579619+0000","last_became_peered":"2023-07-14T19:02:30.579619+0000","last_unstale":"2023-07-14T19:03:31.281462+0000","last_undegraded":"2023-07-14T19:03:31.281462+0000","last_fullsized":"2023-07-14T19:03:31.281462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:51:23.438941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218216+0000","last_change":"2023-07-14T19:03:31.218216+0000","last_active":"2023-07-14T19:03:31.218216+0000","last_peered":"2023-07-14T19:03:31.218216+0000","last_clean":"2023-07-14T19:03:31.218216+0000","last_became_active":"2023-07-14T19:03:31.217871+0000","last_became_peered":"2023-07-14T19:03:31.217871+0000","last_unstale":"2023-07-14T19:03:31.218216+0000","last_undegraded":"2023-07-14T19:03:31.218216+0000","last_fullsized":"2023-07-14T19:03:31.218216+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:56:32.678369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212386+0000","last_change":"2023-07-14T19:02:30.583662+0000","last_active":"2023-07-14T19:03:31.212386+0000","last_peered":"2023-07-14T19:03:31.212386+0000","last_clean":"2023-07-14T19:03:31.212386+0000","last_became_active":"2023-07-14T19:02:30.583352+0000","last_became_peered":"2023-07-14T19:02:30.583352+0000","last_unstale":"2023-07-14T19:03:31.212386+0000","last_undegraded":"2023-07-14T19:03:31.212386+0000","last_fullsized":"2023-07-14T19:03:31.212386+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T07:01:13.319863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213712+0000","last_change":"2023-07-14T19:02:30.567685+0000","last_active":"2023-07-14T19:03:31.213712+0000","last_peered":"2023-07-14T19:03:31.213712+0000","last_clean":"2023-07-14T19:03:31.213712+0000","last_became_active":"2023-07-14T19:02:30.566674+0000","last_became_peered":"2023-07-14T19:02:30.566674+0000","last_unstale":"2023-07-14T19:03:31.213712+0000","last_undegraded":"2023-07-14T19:03:31.213712+0000","last_fullsized":"2023-07-14T19:03:31.213712+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:41:27.460945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210143+0000","last_change":"2023-07-14T19:02:30.558012+0000","last_active":"2023-07-14T19:03:31.210143+0000","last_peered":"2023-07-14T19:03:31.210143+0000","last_clean":"2023-07-14T19:03:31.210143+0000","last_became_active":"2023-07-14T19:02:30.557675+0000","last_became_peered":"2023-07-14T19:02:30.557675+0000","last_unstale":"2023-07-14T19:03:31.210143+0000","last_undegraded":"2023-07-14T19:03:31.210143+0000","last_fullsized":"2023-07-14T19:03:31.210143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:55:41.257017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207948+0000","last_change":"2023-07-14T19:02:30.570341+0000","last_active":"2023-07-14T19:03:31.207948+0000","last_peered":"2023-07-14T19:03:31.207948+0000","last_clean":"2023-07-14T19:03:31.207948+0000","last_became_active":"2023-07-14T19:02:30.570091+0000","last_became_peered":"2023-07-14T19:02:30.570091+0000","last_unstale":"2023-07-14T19:03:31.207948+0000","last_undegraded":"2023-07-14T19:03:31.207948+0000","last_fullsized":"2023-07-14T19:03:31.207948+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:52:13.273335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207886+0000","last_change":"2023-07-14T19:02:30.573746+0000","last_active":"2023-07-14T19:03:31.207886+0000","last_peered":"2023-07-14T19:03:31.207886+0000","last_clean":"2023-07-14T19:03:31.207886+0000","last_became_active":"2023-07-14T19:02:30.573307+0000","last_became_peered":"2023-07-14T19:02:30.573307+0000","last_unstale":"2023-07-14T19:03:31.207886+0000","last_undegraded":"2023-07-14T19:03:31.207886+0000","last_fullsized":"2023-07-14T19:03:31.207886+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:18:49.299962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.280974+0000","last_change":"2023-07-14T19:02:30.578352+0000","last_active":"2023-07-14T19:03:31.280974+0000","last_peered":"2023-07-14T19:03:31.280974+0000","last_clean":"2023-07-14T19:03:31.280974+0000","last_became_active":"2023-07-14T19:02:30.577631+0000","last_became_peered":"2023-07-14T19:02:30.577631+0000","last_unstale":"2023-07-14T19:03:31.280974+0000","last_undegraded":"2023-07-14T19:03:31.280974+0000","last_fullsized":"2023-07-14T19:03:31.280974+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:07:01.573189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209912+0000","last_change":"2023-07-14T19:02:30.577813+0000","last_active":"2023-07-14T19:03:31.209912+0000","last_peered":"2023-07-14T19:03:31.209912+0000","last_clean":"2023-07-14T19:03:31.209912+0000","last_became_active":"2023-07-14T19:02:30.577482+0000","last_became_peered":"2023-07-14T19:02:30.577482+0000","last_unstale":"2023-07-14T19:03:31.209912+0000","last_undegraded":"2023-07-14T19:03:31.209912+0000","last_fullsized":"2023-07-14T19:03:31.209912+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:11:33.046274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210222+0000","last_change":"2023-07-14T19:02:30.583353+0000","last_active":"2023-07-14T19:03:31.210222+0000","last_peered":"2023-07-14T19:03:31.210222+0000","last_clean":"2023-07-14T19:03:31.210222+0000","last_became_active":"2023-07-14T19:02:30.583037+0000","last_became_peered":"2023-07-14T19:02:30.583037+0000","last_unstale":"2023-07-14T19:03:31.210222+0000","last_undegraded":"2023-07-14T19:03:31.210222+0000","last_fullsized":"2023-07-14T19:03:31.210222+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:43:40.021757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214491+0000","last_change":"2023-07-14T19:02:30.569788+0000","last_active":"2023-07-14T19:03:31.214491+0000","last_peered":"2023-07-14T19:03:31.214491+0000","last_clean":"2023-07-14T19:03:31.214491+0000","last_became_active":"2023-07-14T19:02:30.569461+0000","last_became_peered":"2023-07-14T19:02:30.569461+0000","last_unstale":"2023-07-14T19:03:31.214491+0000","last_undegraded":"2023-07-14T19:03:31.214491+0000","last_fullsized":"2023-07-14T19:03:31.214491+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:34:49.917455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214902+0000","last_change":"2023-07-14T19:02:34.585220+0000","last_active":"2023-07-14T19:03:31.214902+0000","last_peered":"2023-07-14T19:03:31.214902+0000","last_clean":"2023-07-14T19:03:31.214902+0000","last_became_active":"2023-07-14T19:02:34.584771+0000","last_became_peered":"2023-07-14T19:02:34.584771+0000","last_unstale":"2023-07-14T19:03:31.214902+0000","last_undegraded":"2023-07-14T19:03:31.214902+0000","last_fullsized":"2023-07-14T19:03:31.214902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:58:38.451682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'9","reported_seq":36,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210698+0000","last_change":"2023-07-14T19:02:29.938571+0000","last_active":"2023-07-14T19:03:31.210698+0000","last_peered":"2023-07-14T19:03:31.210698+0000","last_clean":"2023-07-14T19:03:31.210698+0000","last_became_active":"2023-07-14T19:02:29.938270+0000","last_became_peered":"2023-07-14T19:02:29.938270+0000","last_unstale":"2023-07-14T19:03:31.210698+0000","last_undegraded":"2023-07-14T19:03:31.210698+0000","last_fullsized":"2023-07-14T19:03:31.210698+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:26:02.148507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214143+0000","last_change":"2023-07-14T19:02:30.575894+0000","last_active":"2023-07-14T19:03:31.214143+0000","last_peered":"2023-07-14T19:03:31.214143+0000","last_clean":"2023-07-14T19:03:31.214143+0000","last_became_active":"2023-07-14T19:02:30.575607+0000","last_became_peered":"2023-07-14T19:02:30.575607+0000","last_unstale":"2023-07-14T19:03:31.214143+0000","last_undegraded":"2023-07-14T19:03:31.214143+0000","last_fullsized":"2023-07-14T19:03:31.214143+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:37.751234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218192+0000","last_change":"2023-07-14T19:03:31.218192+0000","last_active":"2023-07-14T19:03:31.218192+0000","last_peered":"2023-07-14T19:03:31.218192+0000","last_clean":"2023-07-14T19:03:31.218192+0000","last_became_active":"2023-07-14T19:03:31.217708+0000","last_became_peered":"2023-07-14T19:03:31.217708+0000","last_unstale":"2023-07-14T19:03:31.218192+0000","last_undegraded":"2023-07-14T19:03:31.218192+0000","last_fullsized":"2023-07-14T19:03:31.218192+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:57:40.671215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,8,3],"acting":[4,6,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214160+0000","last_change":"2023-07-14T19:02:29.944136+0000","last_active":"2023-07-14T19:03:31.214160+0000","last_peered":"2023-07-14T19:03:31.214160+0000","last_clean":"2023-07-14T19:03:31.214160+0000","last_became_active":"2023-07-14T19:02:29.943751+0000","last_became_peered":"2023-07-14T19:02:29.943751+0000","last_unstale":"2023-07-14T19:03:31.214160+0000","last_undegraded":"2023-07-14T19:03:31.214160+0000","last_fullsized":"2023-07-14T19:03:31.214160+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:43:53.346383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208280+0000","last_change":"2023-07-14T19:02:30.583156+0000","last_active":"2023-07-14T19:03:31.208280+0000","last_peered":"2023-07-14T19:03:31.208280+0000","last_clean":"2023-07-14T19:03:31.208280+0000","last_became_active":"2023-07-14T19:02:30.582794+0000","last_became_peered":"2023-07-14T19:02:30.582794+0000","last_unstale":"2023-07-14T19:03:31.208280+0000","last_undegraded":"2023-07-14T19:03:31.208280+0000","last_fullsized":"2023-07-14T19:03:31.208280+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:47:47.558145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214058+0000","last_change":"2023-07-14T19:02:34.590788+0000","last_active":"2023-07-14T19:03:31.214058+0000","last_peered":"2023-07-14T19:03:31.214058+0000","last_clean":"2023-07-14T19:03:31.214058+0000","last_became_active":"2023-07-14T19:02:34.590385+0000","last_became_peered":"2023-07-14T19:02:34.590385+0000","last_unstale":"2023-07-14T19:03:31.214058+0000","last_undegraded":"2023-07-14T19:03:31.214058+0000","last_fullsized":"2023-07-14T19:03:31.214058+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:35.914846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"85'4","reported_seq":42,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.222715+0000","last_change":"2023-07-14T19:03:31.222715+0000","last_active":"2023-07-14T19:03:31.222715+0000","last_peered":"2023-07-14T19:03:31.222715+0000","last_clean":"2023-07-14T19:03:31.222715+0000","last_became_active":"2023-07-14T19:03:31.216343+0000","last_became_peered":"2023-07-14T19:03:31.216343+0000","last_unstale":"2023-07-14T19:03:31.222715+0000","last_undegraded":"2023-07-14T19:03:31.222715+0000","last_fullsized":"2023-07-14T19:03:31.222715+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:10.973001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281375+0000","last_change":"2023-07-14T19:02:30.582746+0000","last_active":"2023-07-14T19:03:31.281375+0000","last_peered":"2023-07-14T19:03:31.281375+0000","last_clean":"2023-07-14T19:03:31.281375+0000","last_became_active":"2023-07-14T19:02:30.582426+0000","last_became_peered":"2023-07-14T19:02:30.582426+0000","last_unstale":"2023-07-14T19:03:31.281375+0000","last_undegraded":"2023-07-14T19:03:31.281375+0000","last_fullsized":"2023-07-14T19:03:31.281375+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:16:53.120645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212948+0000","last_change":"2023-07-14T19:02:34.589966+0000","last_active":"2023-07-14T19:03:31.212948+0000","last_peered":"2023-07-14T19:03:31.212948+0000","last_clean":"2023-07-14T19:03:31.212948+0000","last_became_active":"2023-07-14T19:02:34.589544+0000","last_became_peered":"2023-07-14T19:02:34.589544+0000","last_unstale":"2023-07-14T19:03:31.212948+0000","last_undegraded":"2023-07-14T19:03:31.212948+0000","last_fullsized":"2023-07-14T19:03:31.212948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:10:08.813030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"80'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210414+0000","last_change":"2023-07-14T19:02:29.935507+0000","last_active":"2023-07-14T19:03:31.210414+0000","last_peered":"2023-07-14T19:03:31.210414+0000","last_clean":"2023-07-14T19:03:31.210414+0000","last_became_active":"2023-07-14T19:02:29.935160+0000","last_became_peered":"2023-07-14T19:02:29.935160+0000","last_unstale":"2023-07-14T19:03:31.210414+0000","last_undegraded":"2023-07-14T19:03:31.210414+0000","last_fullsized":"2023-07-14T19:03:31.210414+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:28:02.794267+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219342+0000","last_change":"2023-07-14T19:02:30.578077+0000","last_active":"2023-07-14T19:03:31.219342+0000","last_peered":"2023-07-14T19:03:31.219342+0000","last_clean":"2023-07-14T19:03:31.219342+0000","last_became_active":"2023-07-14T19:02:30.577764+0000","last_became_peered":"2023-07-14T19:02:30.577764+0000","last_unstale":"2023-07-14T19:03:31.219342+0000","last_undegraded":"2023-07-14T19:03:31.219342+0000","last_fullsized":"2023-07-14T19:03:31.219342+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:40:14.188240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214161+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:03:31.214161+0000","last_peered":"2023-07-14T19:03:31.214161+0000","last_clean":"2023-07-14T19:03:31.214161+0000","last_became_active":"2023-07-14T19:02:34.583408+0000","last_became_peered":"2023-07-14T19:02:34.583408+0000","last_unstale":"2023-07-14T19:03:31.214161+0000","last_undegraded":"2023-07-14T19:03:31.214161+0000","last_fullsized":"2023-07-14T19:03:31.214161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:36:10.318901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'6","reported_seq":92,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:45.698469+0000","last_change":"2023-07-14T19:02:29.943899+0000","last_active":"2023-07-14T19:03:45.698469+0000","last_peered":"2023-07-14T19:03:45.698469+0000","last_clean":"2023-07-14T19:03:45.698469+0000","last_became_active":"2023-07-14T19:02:29.943585+0000","last_became_peered":"2023-07-14T19:02:29.943585+0000","last_unstale":"2023-07-14T19:03:45.698469+0000","last_undegraded":"2023-07-14T19:03:45.698469+0000","last_fullsized":"2023-07-14T19:03:45.698469+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:26:29.940165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":964,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":60,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210875+0000","last_change":"2023-07-14T19:02:30.567147+0000","last_active":"2023-07-14T19:03:31.210875+0000","last_peered":"2023-07-14T19:03:31.210875+0000","last_clean":"2023-07-14T19:03:31.210875+0000","last_became_active":"2023-07-14T19:02:30.566792+0000","last_became_peered":"2023-07-14T19:02:30.566792+0000","last_unstale":"2023-07-14T19:03:31.210875+0000","last_undegraded":"2023-07-14T19:03:31.210875+0000","last_fullsized":"2023-07-14T19:03:31.210875+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:00:27.150739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214266+0000","last_change":"2023-07-14T19:02:34.579561+0000","last_active":"2023-07-14T19:03:31.214266+0000","last_peered":"2023-07-14T19:03:31.214266+0000","last_clean":"2023-07-14T19:03:31.214266+0000","last_became_active":"2023-07-14T19:02:34.579099+0000","last_became_peered":"2023-07-14T19:02:34.579099+0000","last_unstale":"2023-07-14T19:03:31.214266+0000","last_undegraded":"2023-07-14T19:03:31.214266+0000","last_fullsized":"2023-07-14T19:03:31.214266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:25:32.422783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"85'3","reported_seq":39,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.284881+0000","last_change":"2023-07-14T19:03:31.284881+0000","last_active":"2023-07-14T19:03:31.284881+0000","last_peered":"2023-07-14T19:03:31.284881+0000","last_clean":"2023-07-14T19:03:31.284881+0000","last_became_active":"2023-07-14T19:03:31.282061+0000","last_became_peered":"2023-07-14T19:03:31.282061+0000","last_unstale":"2023-07-14T19:03:31.284881+0000","last_undegraded":"2023-07-14T19:03:31.284881+0000","last_fullsized":"2023-07-14T19:03:31.284881+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:05:32.928291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,6],"acting":[10,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.220085+0000","last_change":"2023-07-14T19:02:30.575925+0000","last_active":"2023-07-14T19:03:31.220085+0000","last_peered":"2023-07-14T19:03:31.220085+0000","last_clean":"2023-07-14T19:03:31.220085+0000","last_became_active":"2023-07-14T19:02:30.575583+0000","last_became_peered":"2023-07-14T19:02:30.575583+0000","last_unstale":"2023-07-14T19:03:31.220085+0000","last_undegraded":"2023-07-14T19:03:31.220085+0000","last_fullsized":"2023-07-14T19:03:31.220085+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:11:02.245125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209772+0000","last_change":"2023-07-14T19:02:34.587094+0000","last_active":"2023-07-14T19:03:31.209772+0000","last_peered":"2023-07-14T19:03:31.209772+0000","last_clean":"2023-07-14T19:03:31.209772+0000","last_became_active":"2023-07-14T19:02:34.586650+0000","last_became_peered":"2023-07-14T19:02:34.586650+0000","last_unstale":"2023-07-14T19:03:31.209772+0000","last_undegraded":"2023-07-14T19:03:31.209772+0000","last_fullsized":"2023-07-14T19:03:31.209772+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:29:57.546314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209853+0000","last_change":"2023-07-14T19:02:29.934651+0000","last_active":"2023-07-14T19:03:31.209853+0000","last_peered":"2023-07-14T19:03:31.209853+0000","last_clean":"2023-07-14T19:03:31.209853+0000","last_became_active":"2023-07-14T19:02:29.934347+0000","last_became_peered":"2023-07-14T19:02:29.934347+0000","last_unstale":"2023-07-14T19:03:31.209853+0000","last_undegraded":"2023-07-14T19:03:31.209853+0000","last_fullsized":"2023-07-14T19:03:31.209853+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:53:44.964924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213264+0000","last_change":"2023-07-14T19:02:30.571680+0000","last_active":"2023-07-14T19:03:31.213264+0000","last_peered":"2023-07-14T19:03:31.213264+0000","last_clean":"2023-07-14T19:03:31.213264+0000","last_became_active":"2023-07-14T19:02:30.571367+0000","last_became_peered":"2023-07-14T19:02:30.571367+0000","last_unstale":"2023-07-14T19:03:31.213264+0000","last_undegraded":"2023-07-14T19:03:31.213264+0000","last_fullsized":"2023-07-14T19:03:31.213264+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:53.039270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208272+0000","last_change":"2023-07-14T19:02:34.577642+0000","last_active":"2023-07-14T19:03:31.208272+0000","last_peered":"2023-07-14T19:03:31.208272+0000","last_clean":"2023-07-14T19:03:31.208272+0000","last_became_active":"2023-07-14T19:02:34.577232+0000","last_became_peered":"2023-07-14T19:02:34.577232+0000","last_unstale":"2023-07-14T19:03:31.208272+0000","last_undegraded":"2023-07-14T19:03:31.208272+0000","last_fullsized":"2023-07-14T19:03:31.208272+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:01:19.952531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218302+0000","last_change":"2023-07-14T19:02:29.940688+0000","last_active":"2023-07-14T19:03:31.218302+0000","last_peered":"2023-07-14T19:03:31.218302+0000","last_clean":"2023-07-14T19:03:31.218302+0000","last_became_active":"2023-07-14T19:02:29.940444+0000","last_became_peered":"2023-07-14T19:02:29.940444+0000","last_unstale":"2023-07-14T19:03:31.218302+0000","last_undegraded":"2023-07-14T19:03:31.218302+0000","last_fullsized":"2023-07-14T19:03:31.218302+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:09:37.769743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210635+0000","last_change":"2023-07-14T19:02:30.568127+0000","last_active":"2023-07-14T19:03:31.210635+0000","last_peered":"2023-07-14T19:03:31.210635+0000","last_clean":"2023-07-14T19:03:31.210635+0000","last_became_active":"2023-07-14T19:02:30.567811+0000","last_became_peered":"2023-07-14T19:02:30.567811+0000","last_unstale":"2023-07-14T19:03:31.210635+0000","last_undegraded":"2023-07-14T19:03:31.210635+0000","last_fullsized":"2023-07-14T19:03:31.210635+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:08:56.545251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213502+0000","last_change":"2023-07-14T19:02:34.583844+0000","last_active":"2023-07-14T19:03:31.213502+0000","last_peered":"2023-07-14T19:03:31.213502+0000","last_clean":"2023-07-14T19:03:31.213502+0000","last_became_active":"2023-07-14T19:02:34.583480+0000","last_became_peered":"2023-07-14T19:02:34.583480+0000","last_unstale":"2023-07-14T19:03:31.213502+0000","last_undegraded":"2023-07-14T19:03:31.213502+0000","last_fullsized":"2023-07-14T19:03:31.213502+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:26.819356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'3","reported_seq":82,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:41.691911+0000","last_change":"2023-07-14T19:02:29.938459+0000","last_active":"2023-07-14T19:03:41.691911+0000","last_peered":"2023-07-14T19:03:41.691911+0000","last_clean":"2023-07-14T19:03:41.691911+0000","last_became_active":"2023-07-14T19:02:29.938212+0000","last_became_peered":"2023-07-14T19:02:29.938212+0000","last_unstale":"2023-07-14T19:03:41.691911+0000","last_undegraded":"2023-07-14T19:03:41.691911+0000","last_fullsized":"2023-07-14T19:03:41.691911+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:02:34.199818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":53,"num_read_kb":53,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281471+0000","last_change":"2023-07-14T19:02:30.586345+0000","last_active":"2023-07-14T19:03:31.281471+0000","last_peered":"2023-07-14T19:03:31.281471+0000","last_clean":"2023-07-14T19:03:31.281471+0000","last_became_active":"2023-07-14T19:02:30.585949+0000","last_became_peered":"2023-07-14T19:02:30.585949+0000","last_unstale":"2023-07-14T19:03:31.281471+0000","last_undegraded":"2023-07-14T19:03:31.281471+0000","last_fullsized":"2023-07-14T19:03:31.281471+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:50:32.754337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210309+0000","last_change":"2023-07-14T19:02:34.582922+0000","last_active":"2023-07-14T19:03:31.210309+0000","last_peered":"2023-07-14T19:03:31.210309+0000","last_clean":"2023-07-14T19:03:31.210309+0000","last_became_active":"2023-07-14T19:02:34.582146+0000","last_became_peered":"2023-07-14T19:02:34.582146+0000","last_unstale":"2023-07-14T19:03:31.210309+0000","last_undegraded":"2023-07-14T19:03:31.210309+0000","last_fullsized":"2023-07-14T19:03:31.210309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:32:15.121038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212755+0000","last_change":"2023-07-14T19:02:29.938215+0000","last_active":"2023-07-14T19:03:31.212755+0000","last_peered":"2023-07-14T19:03:31.212755+0000","last_clean":"2023-07-14T19:03:31.212755+0000","last_became_active":"2023-07-14T19:02:29.937889+0000","last_became_peered":"2023-07-14T19:02:29.937889+0000","last_unstale":"2023-07-14T19:03:31.212755+0000","last_undegraded":"2023-07-14T19:03:31.212755+0000","last_fullsized":"2023-07-14T19:03:31.212755+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:01:19.043597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208284+0000","last_change":"2023-07-14T19:02:30.569622+0000","last_active":"2023-07-14T19:03:31.208284+0000","last_peered":"2023-07-14T19:03:31.208284+0000","last_clean":"2023-07-14T19:03:31.208284+0000","last_became_active":"2023-07-14T19:02:30.569201+0000","last_became_peered":"2023-07-14T19:02:30.569201+0000","last_unstale":"2023-07-14T19:03:31.208284+0000","last_undegraded":"2023-07-14T19:03:31.208284+0000","last_fullsized":"2023-07-14T19:03:31.208284+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:53:07.451914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214242+0000","last_change":"2023-07-14T19:02:34.580978+0000","last_active":"2023-07-14T19:03:31.214242+0000","last_peered":"2023-07-14T19:03:31.214242+0000","last_clean":"2023-07-14T19:03:31.214242+0000","last_became_active":"2023-07-14T19:02:34.580503+0000","last_became_peered":"2023-07-14T19:02:34.580503+0000","last_unstale":"2023-07-14T19:03:31.214242+0000","last_undegraded":"2023-07-14T19:03:31.214242+0000","last_fullsized":"2023-07-14T19:03:31.214242+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:58:06.819824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"85'4","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219409+0000","last_change":"2023-07-14T19:02:29.942400+0000","last_active":"2023-07-14T19:03:31.219409+0000","last_peered":"2023-07-14T19:03:31.219409+0000","last_clean":"2023-07-14T19:03:31.219409+0000","last_became_active":"2023-07-14T19:02:29.942080+0000","last_became_peered":"2023-07-14T19:02:29.942080+0000","last_unstale":"2023-07-14T19:03:31.219409+0000","last_undegraded":"2023-07-14T19:03:31.219409+0000","last_fullsized":"2023-07-14T19:03:31.219409+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:09:06.130256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281477+0000","last_change":"2023-07-14T19:02:30.582869+0000","last_active":"2023-07-14T19:03:31.281477+0000","last_peered":"2023-07-14T19:03:31.281477+0000","last_clean":"2023-07-14T19:03:31.281477+0000","last_became_active":"2023-07-14T19:02:30.582558+0000","last_became_peered":"2023-07-14T19:02:30.582558+0000","last_unstale":"2023-07-14T19:03:31.281477+0000","last_undegraded":"2023-07-14T19:03:31.281477+0000","last_fullsized":"2023-07-14T19:03:31.281477+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:51:36.705425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.216738+0000","last_change":"2023-07-14T19:03:31.216738+0000","last_active":"2023-07-14T19:03:31.216738+0000","last_peered":"2023-07-14T19:03:31.216738+0000","last_clean":"2023-07-14T19:03:31.216738+0000","last_became_active":"2023-07-14T19:03:31.216310+0000","last_became_peered":"2023-07-14T19:03:31.216310+0000","last_unstale":"2023-07-14T19:03:31.216738+0000","last_undegraded":"2023-07-14T19:03:31.216738+0000","last_fullsized":"2023-07-14T19:03:31.216738+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:26.359586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,10],"acting":[9,8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"85'30","reported_seq":169,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:42.702013+0000","last_change":"2023-07-14T19:02:29.939314+0000","last_active":"2023-07-14T19:03:42.702013+0000","last_peered":"2023-07-14T19:03:42.702013+0000","last_clean":"2023-07-14T19:03:42.702013+0000","last_became_active":"2023-07-14T19:02:29.939000+0000","last_became_peered":"2023-07-14T19:02:29.939000+0000","last_unstale":"2023-07-14T19:03:42.702013+0000","last_undegraded":"2023-07-14T19:03:42.702013+0000","last_fullsized":"2023-07-14T19:03:42.702013+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:43.443151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":56970,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":113,"num_read_kb":112,"num_write":31,"num_write_kb":69,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210508+0000","last_change":"2023-07-14T19:02:30.572052+0000","last_active":"2023-07-14T19:03:31.210508+0000","last_peered":"2023-07-14T19:03:31.210508+0000","last_clean":"2023-07-14T19:03:31.210508+0000","last_became_active":"2023-07-14T19:02:30.571739+0000","last_became_peered":"2023-07-14T19:02:30.571739+0000","last_unstale":"2023-07-14T19:03:31.210508+0000","last_undegraded":"2023-07-14T19:03:31.210508+0000","last_fullsized":"2023-07-14T19:03:31.210508+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:54:15.509239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219888+0000","last_change":"2023-07-14T19:02:34.576663+0000","last_active":"2023-07-14T19:03:31.219888+0000","last_peered":"2023-07-14T19:03:31.219888+0000","last_clean":"2023-07-14T19:03:31.219888+0000","last_became_active":"2023-07-14T19:02:34.576162+0000","last_became_peered":"2023-07-14T19:02:34.576162+0000","last_unstale":"2023-07-14T19:03:31.219888+0000","last_undegraded":"2023-07-14T19:03:31.219888+0000","last_fullsized":"2023-07-14T19:03:31.219888+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:51:50.099226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"85'6","reported_seq":21,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.225143+0000","last_change":"2023-07-14T19:03:31.225143+0000","last_active":"2023-07-14T19:03:31.225143+0000","last_peered":"2023-07-14T19:03:31.225143+0000","last_clean":"2023-07-14T19:03:31.225143+0000","last_became_active":"2023-07-14T19:03:31.215377+0000","last_became_peered":"2023-07-14T19:03:31.215377+0000","last_unstale":"2023-07-14T19:03:31.225143+0000","last_undegraded":"2023-07-14T19:03:31.225143+0000","last_fullsized":"2023-07-14T19:03:31.225143+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:52:23.915375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209950+0000","last_change":"2023-07-14T19:02:30.573798+0000","last_active":"2023-07-14T19:03:31.209950+0000","last_peered":"2023-07-14T19:03:31.209950+0000","last_clean":"2023-07-14T19:03:31.209950+0000","last_became_active":"2023-07-14T19:02:30.572568+0000","last_became_peered":"2023-07-14T19:02:30.572568+0000","last_unstale":"2023-07-14T19:03:31.209950+0000","last_undegraded":"2023-07-14T19:03:31.209950+0000","last_fullsized":"2023-07-14T19:03:31.209950+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:50.422090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210143+0000","last_change":"2023-07-14T19:02:34.582256+0000","last_active":"2023-07-14T19:03:31.210143+0000","last_peered":"2023-07-14T19:03:31.210143+0000","last_clean":"2023-07-14T19:03:31.210143+0000","last_became_active":"2023-07-14T19:02:34.581799+0000","last_became_peered":"2023-07-14T19:02:34.581799+0000","last_unstale":"2023-07-14T19:03:31.210143+0000","last_undegraded":"2023-07-14T19:03:31.210143+0000","last_fullsized":"2023-07-14T19:03:31.210143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:43:26.965376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219545+0000","last_change":"2023-07-14T19:02:29.943874+0000","last_active":"2023-07-14T19:03:31.219545+0000","last_peered":"2023-07-14T19:03:31.219545+0000","last_clean":"2023-07-14T19:03:31.219545+0000","last_became_active":"2023-07-14T19:02:29.943596+0000","last_became_peered":"2023-07-14T19:02:29.943596+0000","last_unstale":"2023-07-14T19:03:31.219545+0000","last_undegraded":"2023-07-14T19:03:31.219545+0000","last_fullsized":"2023-07-14T19:03:31.219545+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:01:17.878050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211874+0000","last_change":"2023-07-14T19:02:30.572157+0000","last_active":"2023-07-14T19:03:31.211874+0000","last_peered":"2023-07-14T19:03:31.211874+0000","last_clean":"2023-07-14T19:03:31.211874+0000","last_became_active":"2023-07-14T19:02:30.571877+0000","last_became_peered":"2023-07-14T19:02:30.571877+0000","last_unstale":"2023-07-14T19:03:31.211874+0000","last_undegraded":"2023-07-14T19:03:31.211874+0000","last_fullsized":"2023-07-14T19:03:31.211874+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:10:27.779325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210374+0000","last_change":"2023-07-14T19:02:34.588594+0000","last_active":"2023-07-14T19:03:31.210374+0000","last_peered":"2023-07-14T19:03:31.210374+0000","last_clean":"2023-07-14T19:03:31.210374+0000","last_became_active":"2023-07-14T19:02:34.587763+0000","last_became_peered":"2023-07-14T19:02:34.587763+0000","last_unstale":"2023-07-14T19:03:31.210374+0000","last_undegraded":"2023-07-14T19:03:31.210374+0000","last_fullsized":"2023-07-14T19:03:31.210374+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:29:02.399855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'4","reported_seq":90,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:45.699224+0000","last_change":"2023-07-14T19:02:29.939145+0000","last_active":"2023-07-14T19:03:45.699224+0000","last_peered":"2023-07-14T19:03:45.699224+0000","last_clean":"2023-07-14T19:03:45.699224+0000","last_became_active":"2023-07-14T19:02:29.938816+0000","last_became_peered":"2023-07-14T19:02:29.938816+0000","last_unstale":"2023-07-14T19:03:45.699224+0000","last_undegraded":"2023-07-14T19:03:45.699224+0000","last_fullsized":"2023-07-14T19:03:45.699224+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:52:05.245627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":930,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":61,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213189+0000","last_change":"2023-07-14T19:02:30.572694+0000","last_active":"2023-07-14T19:03:31.213189+0000","last_peered":"2023-07-14T19:03:31.213189+0000","last_clean":"2023-07-14T19:03:31.213189+0000","last_became_active":"2023-07-14T19:02:30.572391+0000","last_became_peered":"2023-07-14T19:02:30.572391+0000","last_unstale":"2023-07-14T19:03:31.213189+0000","last_undegraded":"2023-07-14T19:03:31.213189+0000","last_fullsized":"2023-07-14T19:03:31.213189+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:38:18.699532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214930+0000","last_change":"2023-07-14T19:03:31.214930+0000","last_active":"2023-07-14T19:03:31.214930+0000","last_peered":"2023-07-14T19:03:31.214930+0000","last_clean":"2023-07-14T19:03:31.214930+0000","last_became_active":"2023-07-14T19:03:31.214574+0000","last_became_peered":"2023-07-14T19:03:31.214574+0000","last_unstale":"2023-07-14T19:03:31.214930+0000","last_undegraded":"2023-07-14T19:03:31.214930+0000","last_fullsized":"2023-07-14T19:03:31.214930+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:08:17.395228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,8,11],"acting":[6,3,8,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"85'15","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.236699+0000","last_change":"2023-07-14T19:03:31.236699+0000","last_active":"2023-07-14T19:03:31.236699+0000","last_peered":"2023-07-14T19:03:31.236699+0000","last_clean":"2023-07-14T19:03:31.236699+0000","last_became_active":"2023-07-14T19:03:31.216096+0000","last_became_peered":"2023-07-14T19:03:31.216096+0000","last_unstale":"2023-07-14T19:03:31.236699+0000","last_undegraded":"2023-07-14T19:03:31.236699+0000","last_fullsized":"2023-07-14T19:03:31.236699+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:23:44.113371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":112,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.282151+0000","last_change":"2023-07-14T19:03:31.282151+0000","last_active":"2023-07-14T19:03:31.282151+0000","last_peered":"2023-07-14T19:03:31.282151+0000","last_clean":"2023-07-14T19:03:31.282151+0000","last_became_active":"2023-07-14T19:03:31.281763+0000","last_became_peered":"2023-07-14T19:03:31.281763+0000","last_unstale":"2023-07-14T19:03:31.282151+0000","last_undegraded":"2023-07-14T19:03:31.282151+0000","last_fullsized":"2023-07-14T19:03:31.282151+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:21:13.429632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,1],"acting":[5,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218880+0000","last_change":"2023-07-14T19:02:34.584170+0000","last_active":"2023-07-14T19:03:31.218880+0000","last_peered":"2023-07-14T19:03:31.218880+0000","last_clean":"2023-07-14T19:03:31.218880+0000","last_became_active":"2023-07-14T19:02:34.583773+0000","last_became_peered":"2023-07-14T19:02:34.583773+0000","last_unstale":"2023-07-14T19:03:31.218880+0000","last_undegraded":"2023-07-14T19:03:31.218880+0000","last_fullsized":"2023-07-14T19:03:31.218880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:32:31.049038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'2","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210253+0000","last_change":"2023-07-14T19:02:29.938386+0000","last_active":"2023-07-14T19:03:31.210253+0000","last_peered":"2023-07-14T19:03:31.210253+0000","last_clean":"2023-07-14T19:03:31.210253+0000","last_became_active":"2023-07-14T19:02:29.937394+0000","last_became_peered":"2023-07-14T19:02:29.937394+0000","last_unstale":"2023-07-14T19:03:31.210253+0000","last_undegraded":"2023-07-14T19:03:31.210253+0000","last_fullsized":"2023-07-14T19:03:31.210253+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:44:18.573548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210241+0000","last_change":"2023-07-14T19:02:30.572485+0000","last_active":"2023-07-14T19:03:31.210241+0000","last_peered":"2023-07-14T19:03:31.210241+0000","last_clean":"2023-07-14T19:03:31.210241+0000","last_became_active":"2023-07-14T19:02:30.572177+0000","last_became_peered":"2023-07-14T19:02:30.572177+0000","last_unstale":"2023-07-14T19:03:31.210241+0000","last_undegraded":"2023-07-14T19:03:31.210241+0000","last_fullsized":"2023-07-14T19:03:31.210241+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:00:15.417585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210735+0000","last_change":"2023-07-14T19:02:34.582340+0000","last_active":"2023-07-14T19:03:31.210735+0000","last_peered":"2023-07-14T19:03:31.210735+0000","last_clean":"2023-07-14T19:03:31.210735+0000","last_became_active":"2023-07-14T19:02:34.581936+0000","last_became_peered":"2023-07-14T19:02:34.581936+0000","last_unstale":"2023-07-14T19:03:31.210735+0000","last_undegraded":"2023-07-14T19:03:31.210735+0000","last_fullsized":"2023-07-14T19:03:31.210735+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:21:00.136660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210852+0000","last_change":"2023-07-14T19:02:29.941404+0000","last_active":"2023-07-14T19:03:31.210852+0000","last_peered":"2023-07-14T19:03:31.210852+0000","last_clean":"2023-07-14T19:03:31.210852+0000","last_became_active":"2023-07-14T19:02:29.941052+0000","last_became_peered":"2023-07-14T19:02:29.941052+0000","last_unstale":"2023-07-14T19:03:31.210852+0000","last_undegraded":"2023-07-14T19:03:31.210852+0000","last_fullsized":"2023-07-14T19:03:31.210852+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:23:27.513794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214237+0000","last_change":"2023-07-14T19:02:30.565671+0000","last_active":"2023-07-14T19:03:31.214237+0000","last_peered":"2023-07-14T19:03:31.214237+0000","last_clean":"2023-07-14T19:03:31.214237+0000","last_became_active":"2023-07-14T19:02:30.565375+0000","last_became_peered":"2023-07-14T19:02:30.565375+0000","last_unstale":"2023-07-14T19:03:31.214237+0000","last_undegraded":"2023-07-14T19:03:31.214237+0000","last_fullsized":"2023-07-14T19:03:31.214237+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:12:48.571954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210193+0000","last_change":"2023-07-14T19:02:34.577692+0000","last_active":"2023-07-14T19:03:31.210193+0000","last_peered":"2023-07-14T19:03:31.210193+0000","last_clean":"2023-07-14T19:03:31.210193+0000","last_became_active":"2023-07-14T19:02:34.577329+0000","last_became_peered":"2023-07-14T19:02:34.577329+0000","last_unstale":"2023-07-14T19:03:31.210193+0000","last_undegraded":"2023-07-14T19:03:31.210193+0000","last_fullsized":"2023-07-14T19:03:31.210193+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:31:07.060258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212127+0000","last_change":"2023-07-14T19:02:29.938330+0000","last_active":"2023-07-14T19:03:31.212127+0000","last_peered":"2023-07-14T19:03:31.212127+0000","last_clean":"2023-07-14T19:03:31.212127+0000","last_became_active":"2023-07-14T19:02:29.938022+0000","last_became_peered":"2023-07-14T19:02:29.938022+0000","last_unstale":"2023-07-14T19:03:31.212127+0000","last_undegraded":"2023-07-14T19:03:31.212127+0000","last_fullsized":"2023-07-14T19:03:31.212127+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:11:24.520411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213923+0000","last_change":"2023-07-14T19:02:30.569353+0000","last_active":"2023-07-14T19:03:31.213923+0000","last_peered":"2023-07-14T19:03:31.213923+0000","last_clean":"2023-07-14T19:03:31.213923+0000","last_became_active":"2023-07-14T19:02:30.569037+0000","last_became_peered":"2023-07-14T19:02:30.569037+0000","last_unstale":"2023-07-14T19:03:31.213923+0000","last_undegraded":"2023-07-14T19:03:31.213923+0000","last_fullsized":"2023-07-14T19:03:31.213923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:22:38.723738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210218+0000","last_change":"2023-07-14T19:02:34.580786+0000","last_active":"2023-07-14T19:03:31.210218+0000","last_peered":"2023-07-14T19:03:31.210218+0000","last_clean":"2023-07-14T19:03:31.210218+0000","last_became_active":"2023-07-14T19:02:34.580395+0000","last_became_peered":"2023-07-14T19:02:34.580395+0000","last_unstale":"2023-07-14T19:03:31.210218+0000","last_undegraded":"2023-07-14T19:03:31.210218+0000","last_fullsized":"2023-07-14T19:03:31.210218+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:45:16.010542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218850+0000","last_change":"2023-07-14T19:02:30.574302+0000","last_active":"2023-07-14T19:03:31.218850+0000","last_peered":"2023-07-14T19:03:31.218850+0000","last_clean":"2023-07-14T19:03:31.218850+0000","last_became_active":"2023-07-14T19:02:30.573912+0000","last_became_peered":"2023-07-14T19:02:30.573912+0000","last_unstale":"2023-07-14T19:03:31.218850+0000","last_undegraded":"2023-07-14T19:03:31.218850+0000","last_fullsized":"2023-07-14T19:03:31.218850+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:25:57.430266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'4","reported_seq":85,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:43.694875+0000","last_change":"2023-07-14T19:02:29.942302+0000","last_active":"2023-07-14T19:03:43.694875+0000","last_peered":"2023-07-14T19:03:43.694875+0000","last_clean":"2023-07-14T19:03:43.694875+0000","last_became_active":"2023-07-14T19:02:29.942005+0000","last_became_peered":"2023-07-14T19:02:29.942005+0000","last_unstale":"2023-07-14T19:03:43.694875+0000","last_undegraded":"2023-07-14T19:03:43.694875+0000","last_fullsized":"2023-07-14T19:03:43.694875+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:30:32.223130+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":55,"num_read_kb":55,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212188+0000","last_change":"2023-07-14T19:02:34.578248+0000","last_active":"2023-07-14T19:03:31.212188+0000","last_peered":"2023-07-14T19:03:31.212188+0000","last_clean":"2023-07-14T19:03:31.212188+0000","last_became_active":"2023-07-14T19:02:34.577930+0000","last_became_peered":"2023-07-14T19:02:34.577930+0000","last_unstale":"2023-07-14T19:03:31.212188+0000","last_undegraded":"2023-07-14T19:03:31.212188+0000","last_fullsized":"2023-07-14T19:03:31.212188+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:52:13.074222+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213462+0000","last_change":"2023-07-14T19:02:30.582892+0000","last_active":"2023-07-14T19:03:31.213462+0000","last_peered":"2023-07-14T19:03:31.213462+0000","last_clean":"2023-07-14T19:03:31.213462+0000","last_became_active":"2023-07-14T19:02:30.582588+0000","last_became_peered":"2023-07-14T19:02:30.582588+0000","last_unstale":"2023-07-14T19:03:31.213462+0000","last_undegraded":"2023-07-14T19:03:31.213462+0000","last_fullsized":"2023-07-14T19:03:31.213462+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:06.056870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"85'3","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214203+0000","last_change":"2023-07-14T19:02:29.935857+0000","last_active":"2023-07-14T19:03:31.214203+0000","last_peered":"2023-07-14T19:03:31.214203+0000","last_clean":"2023-07-14T19:03:31.214203+0000","last_became_active":"2023-07-14T19:02:29.935489+0000","last_became_peered":"2023-07-14T19:02:29.935489+0000","last_unstale":"2023-07-14T19:03:31.214203+0000","last_undegraded":"2023-07-14T19:03:31.214203+0000","last_fullsized":"2023-07-14T19:03:31.214203+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:57:08.956235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":340,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212941+0000","last_change":"2023-07-14T19:01:31.967476+0000","last_active":"2023-07-14T19:03:31.212941+0000","last_peered":"2023-07-14T19:03:31.212941+0000","last_clean":"2023-07-14T19:03:31.212941+0000","last_became_active":"2023-07-14T19:01:31.967146+0000","last_became_peered":"2023-07-14T19:01:31.967146+0000","last_unstale":"2023-07-14T19:03:31.212941+0000","last_undegraded":"2023-07-14T19:03:31.212941+0000","last_fullsized":"2023-07-14T19:03:31.212941+0000","mapping_epoch":72,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":73,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T18:58:40.634620+0000","last_clean_scrub_stamp":"2023-07-14T18:58:40.634620+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:40:56.440561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219439+0000","last_change":"2023-07-14T19:02:34.571823+0000","last_active":"2023-07-14T19:03:31.219439+0000","last_peered":"2023-07-14T19:03:31.219439+0000","last_clean":"2023-07-14T19:03:31.219439+0000","last_became_active":"2023-07-14T19:02:34.571359+0000","last_became_peered":"2023-07-14T19:02:34.571359+0000","last_unstale":"2023-07-14T19:03:31.219439+0000","last_undegraded":"2023-07-14T19:03:31.219439+0000","last_fullsized":"2023-07-14T19:03:31.219439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:39:28.423706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209714+0000","last_change":"2023-07-14T19:02:30.573736+0000","last_active":"2023-07-14T19:03:31.209714+0000","last_peered":"2023-07-14T19:03:31.209714+0000","last_clean":"2023-07-14T19:03:31.209714+0000","last_became_active":"2023-07-14T19:02:30.573381+0000","last_became_peered":"2023-07-14T19:02:30.573381+0000","last_unstale":"2023-07-14T19:03:31.209714+0000","last_undegraded":"2023-07-14T19:03:31.209714+0000","last_fullsized":"2023-07-14T19:03:31.209714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:04:09.104343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213067+0000","last_change":"2023-07-14T19:02:29.937660+0000","last_active":"2023-07-14T19:03:31.213067+0000","last_peered":"2023-07-14T19:03:31.213067+0000","last_clean":"2023-07-14T19:03:31.213067+0000","last_became_active":"2023-07-14T19:02:29.937348+0000","last_became_peered":"2023-07-14T19:02:29.937348+0000","last_unstale":"2023-07-14T19:03:31.213067+0000","last_undegraded":"2023-07-14T19:03:31.213067+0000","last_fullsized":"2023-07-14T19:03:31.213067+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:55:06.823754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211142+0000","last_change":"2023-07-14T19:02:34.581534+0000","last_active":"2023-07-14T19:03:31.211142+0000","last_peered":"2023-07-14T19:03:31.211142+0000","last_clean":"2023-07-14T19:03:31.211142+0000","last_became_active":"2023-07-14T19:02:34.580998+0000","last_became_peered":"2023-07-14T19:02:34.580998+0000","last_unstale":"2023-07-14T19:03:31.211142+0000","last_undegraded":"2023-07-14T19:03:31.211142+0000","last_fullsized":"2023-07-14T19:03:31.211142+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:05:30.391146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"85'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210585+0000","last_change":"2023-07-14T19:02:29.942336+0000","last_active":"2023-07-14T19:03:31.210585+0000","last_peered":"2023-07-14T19:03:31.210585+0000","last_clean":"2023-07-14T19:03:31.210585+0000","last_became_active":"2023-07-14T19:02:29.942033+0000","last_became_peered":"2023-07-14T19:02:29.942033+0000","last_unstale":"2023-07-14T19:03:31.210585+0000","last_undegraded":"2023-07-14T19:03:31.210585+0000","last_fullsized":"2023-07-14T19:03:31.210585+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:04:44.877772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210377+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:03:31.210377+0000","last_peered":"2023-07-14T19:03:31.210377+0000","last_clean":"2023-07-14T19:03:31.210377+0000","last_became_active":"2023-07-14T19:02:30.571129+0000","last_became_peered":"2023-07-14T19:02:30.571129+0000","last_unstale":"2023-07-14T19:03:31.210377+0000","last_undegraded":"2023-07-14T19:03:31.210377+0000","last_fullsized":"2023-07-14T19:03:31.210377+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:38:56.156291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213158+0000","last_change":"2023-07-14T19:02:34.580740+0000","last_active":"2023-07-14T19:03:31.213158+0000","last_peered":"2023-07-14T19:03:31.213158+0000","last_clean":"2023-07-14T19:03:31.213158+0000","last_became_active":"2023-07-14T19:02:34.580344+0000","last_became_peered":"2023-07-14T19:02:34.580344+0000","last_unstale":"2023-07-14T19:03:31.213158+0000","last_undegraded":"2023-07-14T19:03:31.213158+0000","last_fullsized":"2023-07-14T19:03:31.213158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:33:02.155617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'5","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210352+0000","last_change":"2023-07-14T19:02:29.941349+0000","last_active":"2023-07-14T19:03:31.210352+0000","last_peered":"2023-07-14T19:03:31.210352+0000","last_clean":"2023-07-14T19:03:31.210352+0000","last_became_active":"2023-07-14T19:02:29.941039+0000","last_became_peered":"2023-07-14T19:02:29.941039+0000","last_unstale":"2023-07-14T19:03:31.210352+0000","last_undegraded":"2023-07-14T19:03:31.210352+0000","last_fullsized":"2023-07-14T19:03:31.210352+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:38.367807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214004+0000","last_change":"2023-07-14T19:02:30.577546+0000","last_active":"2023-07-14T19:03:31.214004+0000","last_peered":"2023-07-14T19:03:31.214004+0000","last_clean":"2023-07-14T19:03:31.214004+0000","last_became_active":"2023-07-14T19:02:30.576833+0000","last_became_peered":"2023-07-14T19:02:30.576833+0000","last_unstale":"2023-07-14T19:03:31.214004+0000","last_undegraded":"2023-07-14T19:03:31.214004+0000","last_fullsized":"2023-07-14T19:03:31.214004+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:00:55.318416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214192+0000","last_change":"2023-07-14T19:02:34.580655+0000","last_active":"2023-07-14T19:03:31.214192+0000","last_peered":"2023-07-14T19:03:31.214192+0000","last_clean":"2023-07-14T19:03:31.214192+0000","last_became_active":"2023-07-14T19:02:34.580186+0000","last_became_peered":"2023-07-14T19:02:34.580186+0000","last_unstale":"2023-07-14T19:03:31.214192+0000","last_undegraded":"2023-07-14T19:03:31.214192+0000","last_fullsized":"2023-07-14T19:03:31.214192+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:17:17.836503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"85'4","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212620+0000","last_change":"2023-07-14T19:02:29.938343+0000","last_active":"2023-07-14T19:03:31.212620+0000","last_peered":"2023-07-14T19:03:31.212620+0000","last_clean":"2023-07-14T19:03:31.212620+0000","last_became_active":"2023-07-14T19:02:29.938053+0000","last_became_peered":"2023-07-14T19:02:29.938053+0000","last_unstale":"2023-07-14T19:03:31.212620+0000","last_undegraded":"2023-07-14T19:03:31.212620+0000","last_fullsized":"2023-07-14T19:03:31.212620+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:50:45.655040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208419+0000","last_change":"2023-07-14T19:02:30.572734+0000","last_active":"2023-07-14T19:03:31.208419+0000","last_peered":"2023-07-14T19:03:31.208419+0000","last_clean":"2023-07-14T19:03:31.208419+0000","last_became_active":"2023-07-14T19:02:30.572436+0000","last_became_peered":"2023-07-14T19:02:30.572436+0000","last_unstale":"2023-07-14T19:03:31.208419+0000","last_undegraded":"2023-07-14T19:03:31.208419+0000","last_fullsized":"2023-07-14T19:03:31.208419+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:11:10.005214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213237+0000","last_change":"2023-07-14T19:02:34.582900+0000","last_active":"2023-07-14T19:03:31.213237+0000","last_peered":"2023-07-14T19:03:31.213237+0000","last_clean":"2023-07-14T19:03:31.213237+0000","last_became_active":"2023-07-14T19:02:34.582493+0000","last_became_peered":"2023-07-14T19:02:34.582493+0000","last_unstale":"2023-07-14T19:03:31.213237+0000","last_undegraded":"2023-07-14T19:03:31.213237+0000","last_fullsized":"2023-07-14T19:03:31.213237+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:00:45.030002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"85'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.280975+0000","last_change":"2023-07-14T19:02:29.941819+0000","last_active":"2023-07-14T19:03:31.280975+0000","last_peered":"2023-07-14T19:03:31.280975+0000","last_clean":"2023-07-14T19:03:31.280975+0000","last_became_active":"2023-07-14T19:02:29.941469+0000","last_became_peered":"2023-07-14T19:02:29.941469+0000","last_unstale":"2023-07-14T19:03:31.280975+0000","last_undegraded":"2023-07-14T19:03:31.280975+0000","last_fullsized":"2023-07-14T19:03:31.280975+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:40:28.479332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.211086+0000","last_change":"2023-07-14T19:02:30.565125+0000","last_active":"2023-07-14T19:03:31.211086+0000","last_peered":"2023-07-14T19:03:31.211086+0000","last_clean":"2023-07-14T19:03:31.211086+0000","last_became_active":"2023-07-14T19:02:30.564820+0000","last_became_peered":"2023-07-14T19:02:30.564820+0000","last_unstale":"2023-07-14T19:03:31.211086+0000","last_undegraded":"2023-07-14T19:03:31.211086+0000","last_fullsized":"2023-07-14T19:03:31.211086+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:16:52.054472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":12,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.221013+0000","last_change":"2023-07-14T19:03:31.220582+0000","last_active":"2023-07-14T19:03:31.221013+0000","last_peered":"2023-07-14T19:03:31.221013+0000","last_clean":"2023-07-14T19:03:31.221013+0000","last_became_active":"2023-07-14T19:03:31.220162+0000","last_became_peered":"2023-07-14T19:03:31.220162+0000","last_unstale":"2023-07-14T19:03:31.221013+0000","last_undegraded":"2023-07-14T19:03:31.221013+0000","last_fullsized":"2023-07-14T19:03:31.221013+0000","mapping_epoch":86,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":87,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:20:00.068361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,2,5],"acting":[10,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.9","version":"85'3","reported_seq":29,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213644+0000","last_change":"2023-07-14T19:02:29.938027+0000","last_active":"2023-07-14T19:03:31.213644+0000","last_peered":"2023-07-14T19:03:31.213644+0000","last_clean":"2023-07-14T19:03:31.213644+0000","last_became_active":"2023-07-14T19:02:29.937690+0000","last_became_peered":"2023-07-14T19:02:29.937690+0000","last_unstale":"2023-07-14T19:03:31.213644+0000","last_undegraded":"2023-07-14T19:03:31.213644+0000","last_fullsized":"2023-07-14T19:03:31.213644+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:36:21.192935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210779+0000","last_change":"2023-07-14T19:02:30.576369+0000","last_active":"2023-07-14T19:03:31.210779+0000","last_peered":"2023-07-14T19:03:31.210779+0000","last_clean":"2023-07-14T19:03:31.210779+0000","last_became_active":"2023-07-14T19:02:30.576113+0000","last_became_peered":"2023-07-14T19:02:30.576113+0000","last_unstale":"2023-07-14T19:03:31.210779+0000","last_undegraded":"2023-07-14T19:03:31.210779+0000","last_fullsized":"2023-07-14T19:03:31.210779+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:45:57.280886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"85'1","reported_seq":18,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213432+0000","last_change":"2023-07-14T19:02:34.583724+0000","last_active":"2023-07-14T19:03:31.213432+0000","last_peered":"2023-07-14T19:03:31.213432+0000","last_clean":"2023-07-14T19:03:31.213432+0000","last_became_active":"2023-07-14T19:02:34.583218+0000","last_became_peered":"2023-07-14T19:02:34.583218+0000","last_unstale":"2023-07-14T19:03:31.213432+0000","last_undegraded":"2023-07-14T19:03:31.213432+0000","last_fullsized":"2023-07-14T19:03:31.213432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:28:38.807293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":123,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"85'1","reported_seq":27,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213965+0000","last_change":"2023-07-14T19:02:29.938908+0000","last_active":"2023-07-14T19:03:31.213965+0000","last_peered":"2023-07-14T19:03:31.213965+0000","last_clean":"2023-07-14T19:03:31.213965+0000","last_became_active":"2023-07-14T19:02:29.938650+0000","last_became_peered":"2023-07-14T19:02:29.938650+0000","last_unstale":"2023-07-14T19:03:31.213965+0000","last_undegraded":"2023-07-14T19:03:31.213965+0000","last_fullsized":"2023-07-14T19:03:31.213965+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:22:03.037314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213600+0000","last_change":"2023-07-14T19:02:30.571870+0000","last_active":"2023-07-14T19:03:31.213600+0000","last_peered":"2023-07-14T19:03:31.213600+0000","last_clean":"2023-07-14T19:03:31.213600+0000","last_became_active":"2023-07-14T19:02:30.571521+0000","last_became_peered":"2023-07-14T19:02:30.571521+0000","last_unstale":"2023-07-14T19:03:31.213600+0000","last_undegraded":"2023-07-14T19:03:31.213600+0000","last_fullsized":"2023-07-14T19:03:31.213600+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:23:47.125420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213270+0000","last_change":"2023-07-14T19:02:34.589900+0000","last_active":"2023-07-14T19:03:31.213270+0000","last_peered":"2023-07-14T19:03:31.213270+0000","last_clean":"2023-07-14T19:03:31.213270+0000","last_became_active":"2023-07-14T19:02:34.589481+0000","last_became_peered":"2023-07-14T19:02:34.589481+0000","last_unstale":"2023-07-14T19:03:31.213270+0000","last_undegraded":"2023-07-14T19:03:31.213270+0000","last_fullsized":"2023-07-14T19:03:31.213270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:34:07.853098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"85'4","reported_seq":31,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212669+0000","last_change":"2023-07-14T19:02:29.940315+0000","last_active":"2023-07-14T19:03:31.212669+0000","last_peered":"2023-07-14T19:03:31.212669+0000","last_clean":"2023-07-14T19:03:31.212669+0000","last_became_active":"2023-07-14T19:02:29.939991+0000","last_became_peered":"2023-07-14T19:02:29.939991+0000","last_unstale":"2023-07-14T19:03:31.212669+0000","last_undegraded":"2023-07-14T19:03:31.212669+0000","last_fullsized":"2023-07-14T19:03:31.212669+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T04:00:30.521480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212329+0000","last_change":"2023-07-14T19:02:30.583959+0000","last_active":"2023-07-14T19:03:31.212329+0000","last_peered":"2023-07-14T19:03:31.212329+0000","last_clean":"2023-07-14T19:03:31.212329+0000","last_became_active":"2023-07-14T19:02:30.583657+0000","last_became_peered":"2023-07-14T19:02:30.583657+0000","last_unstale":"2023-07-14T19:03:31.212329+0000","last_undegraded":"2023-07-14T19:03:31.212329+0000","last_fullsized":"2023-07-14T19:03:31.212329+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T00:04:07.282087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281103+0000","last_change":"2023-07-14T19:02:34.579328+0000","last_active":"2023-07-14T19:03:31.281103+0000","last_peered":"2023-07-14T19:03:31.281103+0000","last_clean":"2023-07-14T19:03:31.281103+0000","last_became_active":"2023-07-14T19:02:34.578904+0000","last_became_peered":"2023-07-14T19:02:34.578904+0000","last_unstale":"2023-07-14T19:03:31.281103+0000","last_undegraded":"2023-07-14T19:03:31.281103+0000","last_fullsized":"2023-07-14T19:03:31.281103+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:38:15.218624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"85'3","reported_seq":30,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214589+0000","last_change":"2023-07-14T19:02:29.940972+0000","last_active":"2023-07-14T19:03:31.214589+0000","last_peered":"2023-07-14T19:03:31.214589+0000","last_clean":"2023-07-14T19:03:31.214589+0000","last_became_active":"2023-07-14T19:02:29.940670+0000","last_became_peered":"2023-07-14T19:02:29.940670+0000","last_unstale":"2023-07-14T19:03:31.214589+0000","last_undegraded":"2023-07-14T19:03:31.214589+0000","last_fullsized":"2023-07-14T19:03:31.214589+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:55:01.997913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212697+0000","last_change":"2023-07-14T19:02:30.583714+0000","last_active":"2023-07-14T19:03:31.212697+0000","last_peered":"2023-07-14T19:03:31.212697+0000","last_clean":"2023-07-14T19:03:31.212697+0000","last_became_active":"2023-07-14T19:02:30.583404+0000","last_became_peered":"2023-07-14T19:02:30.583404+0000","last_unstale":"2023-07-14T19:03:31.212697+0000","last_undegraded":"2023-07-14T19:03:31.212697+0000","last_fullsized":"2023-07-14T19:03:31.212697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:28:55.354431+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.207943+0000","last_change":"2023-07-14T19:02:34.582985+0000","last_active":"2023-07-14T19:03:31.207943+0000","last_peered":"2023-07-14T19:03:31.207943+0000","last_clean":"2023-07-14T19:03:31.207943+0000","last_became_active":"2023-07-14T19:02:34.582606+0000","last_became_peered":"2023-07-14T19:02:34.582606+0000","last_unstale":"2023-07-14T19:03:31.207943+0000","last_undegraded":"2023-07-14T19:03:31.207943+0000","last_fullsized":"2023-07-14T19:03:31.207943+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:55:50.958196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"80'1","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.219531+0000","last_change":"2023-07-14T19:02:29.944064+0000","last_active":"2023-07-14T19:03:31.219531+0000","last_peered":"2023-07-14T19:03:31.219531+0000","last_clean":"2023-07-14T19:03:31.219531+0000","last_became_active":"2023-07-14T19:02:29.943770+0000","last_became_peered":"2023-07-14T19:02:29.943770+0000","last_unstale":"2023-07-14T19:03:31.219531+0000","last_undegraded":"2023-07-14T19:03:31.219531+0000","last_fullsized":"2023-07-14T19:03:31.219531+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:02:39.670163+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213041+0000","last_change":"2023-07-14T19:02:30.580668+0000","last_active":"2023-07-14T19:03:31.213041+0000","last_peered":"2023-07-14T19:03:31.213041+0000","last_clean":"2023-07-14T19:03:31.213041+0000","last_became_active":"2023-07-14T19:02:30.580297+0000","last_became_peered":"2023-07-14T19:02:30.580297+0000","last_unstale":"2023-07-14T19:03:31.213041+0000","last_undegraded":"2023-07-14T19:03:31.213041+0000","last_fullsized":"2023-07-14T19:03:31.213041+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T06:03:37.240639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.217879+0000","last_change":"2023-07-14T19:02:34.578894+0000","last_active":"2023-07-14T19:03:31.217879+0000","last_peered":"2023-07-14T19:03:31.217879+0000","last_clean":"2023-07-14T19:03:31.217879+0000","last_became_active":"2023-07-14T19:02:34.578463+0000","last_became_peered":"2023-07-14T19:02:34.578463+0000","last_unstale":"2023-07-14T19:03:31.217879+0000","last_undegraded":"2023-07-14T19:03:31.217879+0000","last_fullsized":"2023-07-14T19:03:31.217879+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:33.558780+0000","last_clean_scrub_stamp":"2023-07-14T19:02:33.558780+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T22:04:53.046945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"80'2","reported_seq":28,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210215+0000","last_change":"2023-07-14T19:02:29.938430+0000","last_active":"2023-07-14T19:03:31.210215+0000","last_peered":"2023-07-14T19:03:31.210215+0000","last_clean":"2023-07-14T19:03:31.210215+0000","last_became_active":"2023-07-14T19:02:29.938118+0000","last_became_peered":"2023-07-14T19:02:29.938118+0000","last_unstale":"2023-07-14T19:03:31.210215+0000","last_undegraded":"2023-07-14T19:03:31.210215+0000","last_fullsized":"2023-07-14T19:03:31.210215+0000","mapping_epoch":78,"log_start":"0'0","ondisk_log_start":"0'0","created":78,"last_epoch_clean":79,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:28.916397+0000","last_clean_scrub_stamp":"2023-07-14T19:02:28.916397+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:17:59.720559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.212504+0000","last_change":"2023-07-14T19:02:30.575124+0000","last_active":"2023-07-14T19:03:31.212504+0000","last_peered":"2023-07-14T19:03:31.212504+0000","last_clean":"2023-07-14T19:03:31.212504+0000","last_became_active":"2023-07-14T19:02:30.574805+0000","last_became_peered":"2023-07-14T19:02:30.574805+0000","last_unstale":"2023-07-14T19:03:31.212504+0000","last_undegraded":"2023-07-14T19:03:31.212504+0000","last_fullsized":"2023-07-14T19:03:31.212504+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:42:48.292682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210923+0000","last_change":"2023-07-14T19:02:30.573897+0000","last_active":"2023-07-14T19:03:31.210923+0000","last_peered":"2023-07-14T19:03:31.210923+0000","last_clean":"2023-07-14T19:03:31.210923+0000","last_became_active":"2023-07-14T19:02:30.573585+0000","last_became_peered":"2023-07-14T19:02:30.573585+0000","last_unstale":"2023-07-14T19:03:31.210923+0000","last_undegraded":"2023-07-14T19:03:31.210923+0000","last_fullsized":"2023-07-14T19:03:31.210923+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:26:13.564561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215202+0000","last_change":"2023-07-14T19:02:30.572390+0000","last_active":"2023-07-14T19:03:31.215202+0000","last_peered":"2023-07-14T19:03:31.215202+0000","last_clean":"2023-07-14T19:03:31.215202+0000","last_became_active":"2023-07-14T19:02:30.572093+0000","last_became_peered":"2023-07-14T19:02:30.572093+0000","last_unstale":"2023-07-14T19:03:31.215202+0000","last_undegraded":"2023-07-14T19:03:31.215202+0000","last_fullsized":"2023-07-14T19:03:31.215202+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:57:10.086878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213816+0000","last_change":"2023-07-14T19:02:30.572548+0000","last_active":"2023-07-14T19:03:31.213816+0000","last_peered":"2023-07-14T19:03:31.213816+0000","last_clean":"2023-07-14T19:03:31.213816+0000","last_became_active":"2023-07-14T19:02:30.572260+0000","last_became_peered":"2023-07-14T19:02:30.572260+0000","last_unstale":"2023-07-14T19:03:31.213816+0000","last_undegraded":"2023-07-14T19:03:31.213816+0000","last_fullsized":"2023-07-14T19:03:31.213816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:28:55.025634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.208138+0000","last_change":"2023-07-14T19:02:30.569856+0000","last_active":"2023-07-14T19:03:31.208138+0000","last_peered":"2023-07-14T19:03:31.208138+0000","last_clean":"2023-07-14T19:03:31.208138+0000","last_became_active":"2023-07-14T19:02:30.569546+0000","last_became_peered":"2023-07-14T19:02:30.569546+0000","last_unstale":"2023-07-14T19:03:31.208138+0000","last_undegraded":"2023-07-14T19:03:31.208138+0000","last_fullsized":"2023-07-14T19:03:31.208138+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T05:36:43.152189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210350+0000","last_change":"2023-07-14T19:02:30.570836+0000","last_active":"2023-07-14T19:03:31.210350+0000","last_peered":"2023-07-14T19:03:31.210350+0000","last_clean":"2023-07-14T19:03:31.210350+0000","last_became_active":"2023-07-14T19:02:30.569675+0000","last_became_peered":"2023-07-14T19:02:30.569675+0000","last_unstale":"2023-07-14T19:03:31.210350+0000","last_undegraded":"2023-07-14T19:03:31.210350+0000","last_fullsized":"2023-07-14T19:03:31.210350+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T03:26:19.766316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.209916+0000","last_change":"2023-07-14T19:02:30.566846+0000","last_active":"2023-07-14T19:03:31.209916+0000","last_peered":"2023-07-14T19:03:31.209916+0000","last_clean":"2023-07-14T19:03:31.209916+0000","last_became_active":"2023-07-14T19:02:30.566525+0000","last_became_peered":"2023-07-14T19:02:30.566525+0000","last_unstale":"2023-07-14T19:03:31.209916+0000","last_undegraded":"2023-07-14T19:03:31.209916+0000","last_fullsized":"2023-07-14T19:03:31.209916+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T21:12:37.744089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.215127+0000","last_change":"2023-07-14T19:02:30.579680+0000","last_active":"2023-07-14T19:03:31.215127+0000","last_peered":"2023-07-14T19:03:31.215127+0000","last_clean":"2023-07-14T19:03:31.215127+0000","last_became_active":"2023-07-14T19:02:30.579379+0000","last_became_peered":"2023-07-14T19:02:30.579379+0000","last_unstale":"2023-07-14T19:03:31.215127+0000","last_undegraded":"2023-07-14T19:03:31.215127+0000","last_fullsized":"2023-07-14T19:03:31.215127+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:00:25.682516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.214145+0000","last_change":"2023-07-14T19:02:30.577722+0000","last_active":"2023-07-14T19:03:31.214145+0000","last_peered":"2023-07-14T19:03:31.214145+0000","last_clean":"2023-07-14T19:03:31.214145+0000","last_became_active":"2023-07-14T19:02:30.577247+0000","last_became_peered":"2023-07-14T19:02:30.577247+0000","last_unstale":"2023-07-14T19:03:31.214145+0000","last_undegraded":"2023-07-14T19:03:31.214145+0000","last_fullsized":"2023-07-14T19:03:31.214145+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T23:19:14.172473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.218436+0000","last_change":"2023-07-14T19:02:30.571052+0000","last_active":"2023-07-14T19:03:31.218436+0000","last_peered":"2023-07-14T19:03:31.218436+0000","last_clean":"2023-07-14T19:03:31.218436+0000","last_became_active":"2023-07-14T19:02:30.570722+0000","last_became_peered":"2023-07-14T19:02:30.570722+0000","last_unstale":"2023-07-14T19:03:31.218436+0000","last_undegraded":"2023-07-14T19:03:31.218436+0000","last_fullsized":"2023-07-14T19:03:31.218436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T19:50:37.718571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.213772+0000","last_change":"2023-07-14T19:02:30.575685+0000","last_active":"2023-07-14T19:03:31.213772+0000","last_peered":"2023-07-14T19:03:31.213772+0000","last_clean":"2023-07-14T19:03:31.213772+0000","last_became_active":"2023-07-14T19:02:30.575379+0000","last_became_peered":"2023-07-14T19:02:30.575379+0000","last_unstale":"2023-07-14T19:03:31.213772+0000","last_undegraded":"2023-07-14T19:03:31.213772+0000","last_fullsized":"2023-07-14T19:03:31.213772+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:14:01.199281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.281456+0000","last_change":"2023-07-14T19:02:30.582450+0000","last_active":"2023-07-14T19:03:31.281456+0000","last_peered":"2023-07-14T19:03:31.281456+0000","last_clean":"2023-07-14T19:03:31.281456+0000","last_became_active":"2023-07-14T19:02:30.582074+0000","last_became_peered":"2023-07-14T19:02:30.582074+0000","last_unstale":"2023-07-14T19:03:31.281456+0000","last_undegraded":"2023-07-14T19:03:31.281456+0000","last_fullsized":"2023-07-14T19:03:31.281456+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T01:36:50.575652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210265+0000","last_change":"2023-07-14T19:02:30.573042+0000","last_active":"2023-07-14T19:03:31.210265+0000","last_peered":"2023-07-14T19:03:31.210265+0000","last_clean":"2023-07-14T19:03:31.210265+0000","last_became_active":"2023-07-14T19:02:30.572498+0000","last_became_peered":"2023-07-14T19:02:30.572498+0000","last_unstale":"2023-07-14T19:03:31.210265+0000","last_undegraded":"2023-07-14T19:03:31.210265+0000","last_fullsized":"2023-07-14T19:03:31.210265+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-15T20:11:01.947296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":87,"state":"active+clean","last_fresh":"2023-07-14T19:03:31.210278+0000","last_change":"2023-07-14T19:02:30.569441+0000","last_active":"2023-07-14T19:03:31.210278+0000","last_peered":"2023-07-14T19:03:31.210278+0000","last_clean":"2023-07-14T19:03:31.210278+0000","last_became_active":"2023-07-14T19:02:30.569180+0000","last_became_peered":"2023-07-14T19:02:30.569180+0000","last_unstale":"2023-07-14T19:03:31.210278+0000","last_undegraded":"2023-07-14T19:03:31.210278+0000","last_fullsized":"2023-07-14T19:03:31.210278+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-07-14T19:02:29.920433+0000","last_clean_scrub_stamp":"2023-07-14T19:02:29.920433+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-07-16T02:06:24.866934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":246,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":61264,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":354,"num_read_kb":239,"num_write":153,"num_write_kb":107,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":146,"num_keys_recovered":20,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":183792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":76,"seq":326417514523,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":983040,"data_stored":476824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.496}]},{"osd":1,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.93100000000000005,"5min":0.93100000000000005,"15min":0.93100000000000005},"last":0.81899999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.63200000000000001}]},{"osd":2,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.0669999999999999,"5min":1.0669999999999999,"15min":1.0669999999999999},"last":0.70399999999999996}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"min":{"1min":0.505,"5min":0.505,"15min":0.505},"max":{"1min":1.0349999999999999,"5min":1.0349999999999999,"15min":1.0349999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"last":0.57499999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.375,"5min":1.375,"15min":1.375},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.85999999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.6220000000000001},"last":0.68600000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.745,"5min":0.745,"15min":0.745},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.1859999999999999,"5min":1.1859999999999999,"15min":1.1859999999999999},"last":0.73099999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.751},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.995}]},{"osd":9,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.77500000000000002,"5min":0.77500000000000002,"15min":0.77500000000000002},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.72,"5min":1.72,"15min":1.72},"last":0.83899999999999997},{"interface":"front","average":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.3999999999999999,"5min":1.3999999999999999,"15min":1.3999999999999999},"last":0.93000000000000005}]},{"osd":10,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.6819999999999999,"5min":1.6819999999999999,"15min":1.6819999999999999},"last":0.80300000000000005}]}]},{"osd":10,"up_from":69,"seq":296352743454,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28640,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716512,"statfs":{"total":95995035648,"available":95965708288,"internally_reserved":0,"allocated":1654784,"data_stored":1121258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.45600000000000002,"15min":0.45300000000000001},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.66300000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.48699999999999999,"15min":0.48399999999999999},"min":{"1min":0.33700000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.57299999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.63800000000000001,"5min":0.59699999999999998,"15min":0.58999999999999997},"min":{"1min":0.498,"5min":0.42599999999999999,"15min":0.42599999999999999},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.71799999999999997},{"interface":"front","average":{"1min":0.68700000000000006,"5min":0.63100000000000001,"15min":0.621},"min":{"1min":0.48499999999999999,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.59699999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.624,"15min":0.628},"min":{"1min":0.442,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.77500000000000002,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.64200000000000002,"15min":0.65200000000000002},"min":{"1min":0.45400000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79000000000000004,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.61699999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.59999999999999998,"15min":0.60099999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.432},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.80500000000000005,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.55200000000000005}]},{"osd":5,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.68600000000000005,"5min":0.65400000000000003,"15min":0.64900000000000002},"min":{"1min":0.47099999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.70399999999999996,"15min":0.69499999999999995},"min":{"1min":0.56399999999999995,"5min":0.433,"15min":0.433},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.68300000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.70399999999999996,"5min":0.69499999999999995,"15min":0.69299999999999995},"min":{"1min":0.51200000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.89100000000000001,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.70899999999999996,"5min":0.71699999999999997,"15min":0.71899999999999997},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.1519999999999999,"5min":1.1519999999999999,"15min":1.1519999999999999},"last":0.69999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59499999999999997}]},{"osd":8,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.77300000000000002,"5min":0.77800000000000002,"15min":0.77900000000000003},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.75800000000000001},{"interface":"front","average":{"1min":0.75,"5min":0.77000000000000002,"15min":0.77400000000000002},"min":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"max":{"1min":0.96699999999999997,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.79300000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.77800000000000002,"15min":0.78200000000000003},"min":{"1min":0.59599999999999997,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.1970000000000001,"5min":1.1970000000000001,"15min":1.1970000000000001},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.70799999999999996,"5min":0.747,"15min":0.753},"min":{"1min":0.57199999999999995,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.88,"5min":1.226,"15min":1.226},"last":0.76500000000000001}]},{"osd":11,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"min":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.747},{"interface":"front","average":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"min":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.78200000000000003}]}]},{"osd":9,"up_from":64,"seq":274877906979,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":481010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.41699999999999998,"15min":0.43099999999999999},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.42099999999999999,"5min":0.63,"15min":0.63},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.42499999999999999,"15min":0.435},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.53800000000000003,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.42899999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.58599999999999997,"15min":0.59399999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77600000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.61699999999999999,"15min":0.626},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79000000000000004,"5min":0.85699999999999998,"15min":0.85699999999999998},"last":0.47899999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.60399999999999998,"15min":0.61399999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55100000000000005,"5min":0.61699999999999999,"15min":0.628},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.71299999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.67100000000000004}]},{"osd":3,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.47699999999999998,"15min":0.49299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.66300000000000003,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.497},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.48499999999999999,"15min":0.498},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.55400000000000005}]},{"osd":4,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.64400000000000002,"15min":0.65800000000000003},"min":{"1min":0.41099999999999998,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.72499999999999998,"5min":1.008,"15min":1.008},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.57699999999999996,"5min":0.628,"15min":0.63600000000000001},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.746,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.56899999999999995}]},{"osd":5,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.65700000000000003,"15min":0.67100000000000004},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.70499999999999996,"5min":1.026,"15min":1.026},"last":0.68600000000000005},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.65900000000000003,"15min":0.67500000000000004},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.73399999999999999,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.63300000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:02:06 2023","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.96299999999999997,"5min":0.96299999999999997,"15min":0.96299999999999997},"last":0.62},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.59399999999999997}]},{"osd":7,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.69999999999999996,"15min":0.71599999999999997},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":0.80300000000000005,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.70299999999999996,"15min":0.72099999999999997},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.81799999999999995,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46000000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.73699999999999999,"15min":0.755},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":0.81399999999999995,"5min":0.96699999999999997,"15min":0.96699999999999997},"last":0.623},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.72399999999999998,"15min":0.74199999999999999},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.84299999999999997,"5min":0.98399999999999999,"15min":0.98399999999999999},"last":0.76400000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.69399999999999995,"15min":0.70899999999999996},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.746,"5min":0.97099999999999997,"15min":0.97099999999999997},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.63700000000000001,"5min":0.70399999999999996,"15min":0.71499999999999997},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.79400000000000004,"5min":0.96099999999999997,"15min":0.96099999999999997},"last":0.71499999999999997}]},{"osd":11,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.73999999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.65900000000000003}]}]},{"osd":8,"up_from":57,"seq":244813135911,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":476391,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.59799999999999998,"15min":0.60399999999999998},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.88300000000000001,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.56999999999999995,"15min":0.57599999999999996},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.68500000000000005}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68100000000000005,"5min":0.628,"15min":0.61899999999999999},"min":{"1min":0.45500000000000002,"5min":0.442,"15min":0.442},"max":{"1min":0.90900000000000003,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56899999999999995,"15min":0.56799999999999995},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.76300000000000001,"5min":0.90300000000000002,"15min":0.90300000000000002},"last":0.66400000000000003}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.47499999999999998,"15min":0.47399999999999998},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.496},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.47799999999999998,"15min":0.48299999999999998},"min":{"1min":0.28899999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67700000000000005,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.53500000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.61799999999999999,"15min":0.622},"min":{"1min":0.42399999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.83399999999999996,"5min":0.878,"15min":0.878},"last":0.748},{"interface":"front","average":{"1min":0.622,"5min":0.67500000000000004,"15min":0.68400000000000005},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.88500000000000001,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.76200000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.63400000000000001,"15min":0.626},"min":{"1min":0.44600000000000001,"5min":0.432,"15min":0.432},"max":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"last":0.626},{"interface":"front","average":{"1min":0.71999999999999997,"5min":0.68400000000000005,"15min":0.67800000000000005},"min":{"1min":0.41099999999999998,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"last":0.69999999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.56399999999999995,"15min":0.56499999999999995},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.97899999999999998,"5min":0.97899999999999998,"15min":0.97899999999999998},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.66100000000000003,"5min":0.626,"15min":0.62},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.64800000000000002}]},{"osd":6,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.64600000000000002,"15min":0.65500000000000003},"min":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.44800000000000001},"max":{"1min":0.81499999999999995,"5min":0.97799999999999998,"15min":0.97799999999999998},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.72599999999999998,"15min":0.73399999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.94299999999999995,"5min":1.01,"15min":1.01},"last":0.80000000000000004}]},{"osd":7,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.68700000000000006,"15min":0.68700000000000006},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.85499999999999998,"5min":0.871,"15min":0.871},"last":0.77200000000000002},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.0369999999999999,"5min":1.046,"15min":1.046},"last":0.88700000000000001}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.78400000000000003,"5min":0.755,"15min":0.75},"min":{"1min":0.53200000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.036,"5min":1.036,"15min":1.036},"last":0.872},{"interface":"front","average":{"1min":0.68400000000000005,"5min":0.70999999999999996,"15min":0.71399999999999997},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.032,"5min":1.032,"15min":1.032},"last":0.82899999999999996}]},{"osd":10,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.83699999999999997,"5min":0.77600000000000002,"15min":0.76600000000000001},"min":{"1min":0.64600000000000002,"5min":0.48099999999999998,"15min":0.48099999999999998},"max":{"1min":1.016,"5min":1.1140000000000001,"15min":1.1140000000000001},"last":0.85999999999999999},{"interface":"front","average":{"1min":0.88600000000000001,"5min":0.79600000000000004,"15min":0.78100000000000003},"min":{"1min":0.70299999999999996,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.0700000000000001,"5min":1.081,"15min":1.081},"last":0.81699999999999995}]},{"osd":11,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.0489999999999999,"5min":1.0489999999999999,"15min":1.0489999999999999},"last":0.91000000000000003},{"interface":"front","average":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"min":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.038,"5min":1.038,"15min":1.038},"last":0.84799999999999998}]}]},{"osd":7,"up_from":50,"seq":214748364844,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28564,"kb_used_data":1540,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716588,"statfs":{"total":95995035648,"available":95965786112,"internally_reserved":0,"allocated":1576960,"data_stored":1072763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.52900000000000003,"15min":0.52600000000000002},"min":{"1min":0.39400000000000002,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.59199999999999997}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.40600000000000003,"15min":0.39200000000000002},"min":{"1min":0.33300000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.56999999999999995,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.42499999999999999,"15min":0.40899999999999997},"min":{"1min":0.32700000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.65700000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.63200000000000001,"5min":0.57399999999999995,"15min":0.54900000000000004},"min":{"1min":0.378,"5min":0.31,"15min":0.31},"max":{"1min":0.82599999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.622,"5min":0.56299999999999994,"15min":0.54500000000000004},"min":{"1min":0.41099999999999998,"5min":0.37,"15min":0.37},"max":{"1min":0.78900000000000003,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.53900000000000003}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.54500000000000004,"15min":0.53100000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.71699999999999997,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.58399999999999996},"min":{"1min":0.40699999999999997,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.79500000000000004,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.67000000000000004}]},{"osd":4,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.47699999999999998,"15min":0.45900000000000002},"min":{"1min":0.38300000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.497},{"interface":"front","average":{"1min":0.497,"5min":0.49399999999999999,"15min":0.48599999999999999},"min":{"1min":0.37,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.69999999999999996,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64800000000000002}]},{"osd":5,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.68899999999999995,"5min":0.627,"15min":0.61699999999999999},"min":{"1min":0.42599999999999999,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.70199999999999996,"5min":0.63200000000000001,"15min":0.60799999999999998},"min":{"1min":0.44,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.87,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.60999999999999999}]},{"osd":6,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.64400000000000002,"15min":0.63},"min":{"1min":0.54100000000000004,"5min":0.48599999999999999,"15min":0.48599999999999999},"max":{"1min":0.85699999999999998,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.65400000000000003,"5min":0.65500000000000003,"15min":0.63900000000000001},"min":{"1min":0.496,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.94899999999999995,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.76800000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.66700000000000004,"15min":0.66500000000000004},"min":{"1min":0.40899999999999997,"5min":0.40899999999999997,"15min":0.40899999999999997},"max":{"1min":0.88700000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.74299999999999999,"5min":0.69899999999999995,"15min":0.69099999999999995},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.68700000000000006}]},{"osd":9,"last update":"Fri Jul 14 19:03:11 2023","interfaces":[{"interface":"back","average":{"1min":0.72699999999999998,"5min":0.74099999999999999,"15min":0.74299999999999999},"min":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"max":{"1min":0.96999999999999997,"5min":0.98899999999999999,"15min":0.98899999999999999},"last":0.745},{"interface":"front","average":{"1min":0.997,"5min":0.76300000000000001,"15min":0.72399999999999998},"min":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"max":{"1min":6.8970000000000002,"5min":6.8970000000000002,"15min":6.8970000000000002},"last":0.78100000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76100000000000001,"5min":0.72899999999999998,"15min":0.72299999999999998},"min":{"1min":0.60599999999999998,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.95299999999999996,"5min":1.0129999999999999,"15min":1.0129999999999999},"last":0.79400000000000004},{"interface":"front","average":{"1min":0.745,"5min":0.69399999999999995,"15min":0.68600000000000005},"min":{"1min":0.64700000000000002,"5min":0.46600000000000003,"15min":0.46600000000000003},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.73299999999999998}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"min":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"max":{"1min":1.0089999999999999,"5min":1.0089999999999999,"15min":1.0089999999999999},"last":0.80900000000000005},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"last":0.755}]}]},{"osd":6,"up_from":45,"seq":193273528368,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":472090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39900000000000002,"15min":0.38700000000000001},"min":{"1min":0.26900000000000002,"5min":0.224,"15min":0.224},"max":{"1min":0.54700000000000004,"5min":1.1439999999999999,"15min":1.1439999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.41999999999999998,"15min":0.41599999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.56200000000000006,"5min":1.591,"15min":1.591},"last":0.33700000000000002}]},{"osd":1,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.58899999999999997,"15min":0.55700000000000005},"min":{"1min":0.41299999999999998,"5min":0.376,"15min":0.376},"max":{"1min":0.83699999999999997,"5min":1.159,"15min":1.159},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.55400000000000005,"15min":0.53900000000000003},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.752,"5min":1.49,"15min":1.49},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.58699999999999997,"15min":0.58899999999999997},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73799999999999999,"5min":1.3680000000000001,"15min":1.3680000000000001},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.59499999999999997,"5min":0.61099999999999999,"15min":0.59299999999999997},"min":{"1min":0.45500000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.77900000000000003,"5min":1.419,"15min":1.419},"last":0.58499999999999996}]},{"osd":3,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.34300000000000003,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.438},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.48199999999999998,"15min":0.48599999999999999},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.60899999999999999,"5min":2.077,"15min":2.077},"last":0.38800000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.59799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42099999999999999,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.85299999999999998,"5min":1.3180000000000001,"15min":1.3180000000000001},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.64600000000000002,"15min":0.624},"min":{"1min":0.45500000000000002,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.80900000000000005,"5min":1.1990000000000001,"15min":1.1990000000000001},"last":0.75800000000000001}]},{"osd":5,"last update":"Fri Jul 14 19:03:06 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59199999999999997,"15min":0.58799999999999997},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.75,"5min":1.3879999999999999,"15min":1.3879999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64700000000000002,"15min":0.65100000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.76200000000000001,"5min":1.2230000000000001,"15min":1.2230000000000001},"last":0.56299999999999994}]},{"osd":7,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.66800000000000004,"15min":0.67900000000000005},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.76300000000000001,"5min":1.2529999999999999,"15min":1.2529999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.75,"15min":0.73099999999999998},"min":{"1min":0.51500000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.86699999999999999,"5min":3.6709999999999998,"15min":3.6709999999999998},"last":0.72799999999999998}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.73099999999999998,"15min":0.73599999999999999},"min":{"1min":0.51200000000000001,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.1000000000000001,"5min":1.1000000000000001,"15min":1.1000000000000001},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.70599999999999996,"15min":0.71499999999999997},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":1.1699999999999999,"5min":1.1699999999999999,"15min":1.1699999999999999},"last":0.61499999999999999}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.77400000000000002,"15min":0.80400000000000005},"min":{"1min":0.42399999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.78900000000000003,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.68200000000000005,"15min":0.70699999999999996},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":1.1240000000000001,"5min":1.5309999999999999,"15min":1.5309999999999999},"last":0.52200000000000002}]},{"osd":10,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.73499999999999999,"15min":0.751},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":0.82499999999999996,"5min":1.2829999999999999,"15min":1.2829999999999999},"last":0.65000000000000002},{"interface":"front","average":{"1min":0.72299999999999998,"5min":0.81100000000000005,"15min":0.82499999999999996},"min":{"1min":0.55700000000000005,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":0.88600000000000001,"5min":1.234,"15min":1.234},"last":0.83899999999999997}]},{"osd":11,"last update":"Fri Jul 14 19:03:35 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"min":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"max":{"1min":0.84199999999999997,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.81299999999999994}]}]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":528970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.52400000000000002,"15min":0.49399999999999999},"min":{"1min":0.42799999999999999,"5min":0.309,"15min":0.309},"max":{"1min":2.3370000000000002,"5min":2.3370000000000002,"15min":2.3370000000000002},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.53000000000000003,"15min":0.51200000000000001},"min":{"1min":0.379,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.2469999999999999,"5min":2.2469999999999999,"15min":2.2469999999999999},"last":0.58499999999999996}]},{"osd":2,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.67600000000000005,"5min":0.55200000000000005,"15min":0.52100000000000002},"min":{"1min":0.40899999999999997,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.626,"5min":0.52500000000000002,"15min":0.5},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.77100000000000002,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.60499999999999998}]},{"osd":3,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.39500000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.88300000000000001,"5min":0.94499999999999995,"15min":0.94499999999999995},"last":0.72899999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.58099999999999996,"15min":0.55700000000000005},"min":{"1min":0.38600000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.81999999999999995,"5min":0.91500000000000004,"15min":0.91500000000000004},"last":0.75700000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:03:25 2023","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.44600000000000001,"15min":0.39100000000000001},"min":{"1min":0.36599999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.44600000000000001,"15min":0.40200000000000002},"min":{"1min":0.45100000000000001,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.498}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.66800000000000004,"5min":0.57099999999999995,"15min":0.53900000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.86899999999999999,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.56599999999999995,"15min":0.54800000000000004},"min":{"1min":0.434,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.85099999999999998,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.70899999999999996}]},{"osd":6,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.61799999999999999,"15min":0.59199999999999997},"min":{"1min":0.53800000000000003,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.94299999999999995,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.626},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.65000000000000002,"15min":0.60199999999999998},"min":{"1min":0.46500000000000002,"5min":0.40500000000000003,"15min":0.40500000000000003},"max":{"1min":4.3209999999999997,"5min":4.3209999999999997,"15min":4.3209999999999997},"last":0.67900000000000005}]},{"osd":7,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.54900000000000004,"15min":0.51700000000000002},"min":{"1min":0.439,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.84599999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.78700000000000003,"5min":0.628,"15min":0.58199999999999996},"min":{"1min":0.52200000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.97399999999999998,"5min":0.97399999999999998,"15min":0.97399999999999998},"last":0.80500000000000005}]},{"osd":8,"last update":"Fri Jul 14 19:02:52 2023","interfaces":[{"interface":"back","average":{"1min":0.80500000000000005,"5min":0.72899999999999998,"15min":0.71599999999999997},"min":{"1min":0.498,"5min":0.47999999999999998,"15min":0.47999999999999998},"max":{"1min":1.0309999999999999,"5min":1.0309999999999999,"15min":1.0309999999999999},"last":0.88700000000000001},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.68999999999999995,"15min":0.67900000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.99299999999999999,"5min":0.996,"15min":0.996},"last":0.81799999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:03:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89200000000000002,"5min":0.79000000000000004,"15min":0.77300000000000002},"min":{"1min":0.54800000000000004,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.85699999999999998},{"interface":"front","average":{"1min":0.80100000000000005,"5min":0.73999999999999999,"15min":0.72999999999999998},"min":{"1min":0.60099999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":1.0900000000000001,"5min":1.0900000000000001,"15min":1.0900000000000001},"last":0.86899999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.82999999999999996,"5min":0.77000000000000002,"15min":0.76000000000000001},"min":{"1min":0.625,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":1.0780000000000001},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.77200000000000002,"5min":0.73199999999999998,"15min":0.72499999999999998},"min":{"1min":0.60699999999999998,"5min":0.51900000000000002,"15min":0.51900000000000002},"max":{"1min":0.90300000000000002,"5min":0.92800000000000005,"15min":0.92800000000000005},"last":0.79000000000000004}]},{"osd":11,"last update":"Fri Jul 14 19:03:32 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.92800000000000005,"15min":0.92800000000000005},"min":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.093,"5min":1.093,"15min":1.093},"last":0.91500000000000004},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":0.93500000000000005,"5min":0.93500000000000005,"15min":0.93500000000000005},"last":0.84299999999999997}]}]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":473205,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.54400000000000004,"15min":0.50700000000000001},"min":{"1min":0.39800000000000002,"5min":0.159,"15min":0.159},"max":{"1min":0.72499999999999998,"5min":0.99199999999999999,"15min":0.99199999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52000000000000002,"15min":0.48999999999999999},"min":{"1min":0.40300000000000002,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.75800000000000001,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.66500000000000004}]},{"osd":2,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.37,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.88200000000000001,"5min":0.88200000000000001,"15min":0.88200000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53000000000000003,"15min":0.504},"min":{"1min":0.39200000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.746,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.64800000000000002}]},{"osd":3,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.831,"5min":1.831,"15min":1.831},"last":0.53400000000000003}]},{"osd":4,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.55200000000000005,"15min":0.52600000000000002},"min":{"1min":0.372,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.68700000000000006,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.54700000000000004,"15min":0.50900000000000001},"min":{"1min":0.43099999999999999,"5min":0.309,"15min":0.309},"max":{"1min":1.399,"5min":1.399,"15min":1.399},"last":0.71799999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.58799999999999997,"15min":0.56299999999999994},"min":{"1min":0.46000000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.91700000000000004,"5min":1.042,"15min":1.042},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56299999999999994,"15min":0.55200000000000005},"min":{"1min":0.41799999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.83699999999999997,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.69699999999999995}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.496,"15min":0.495},"min":{"1min":0.29899999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.59799999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57499999999999996,"15min":0.56899999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72099999999999997,"5min":1.1359999999999999,"15min":1.1359999999999999},"last":0.81399999999999995}]},{"osd":7,"last update":"Fri Jul 14 19:03:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.439,"5min":0.439,"15min":0.439},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.73499999999999999},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.3420000000000001,"5min":1.3420000000000001,"15min":1.3420000000000001},"last":0.80000000000000004}]},{"osd":8,"last update":"Fri Jul 14 19:02:47 2023","interfaces":[{"interface":"back","average":{"1min":0.67400000000000004,"5min":0.70199999999999996,"15min":0.70599999999999996},"min":{"1min":0.46000000000000002,"5min":0.435,"15min":0.435},"max":{"1min":0.97299999999999998,"5min":1.024,"15min":1.024},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.74199999999999999,"15min":0.747},"min":{"1min":0.498,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.026,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.85799999999999998}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.65300000000000002,"15min":0.67700000000000005},"min":{"1min":0.373,"5min":0.372,"15min":0.372},"max":{"1min":0.78200000000000003,"5min":1.204,"15min":1.204},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.72499999999999998,"15min":0.746},"min":{"1min":0.39500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.82199999999999995,"5min":1.3100000000000001,"15min":1.3100000000000001},"last":0.76600000000000001}]},{"osd":10,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.68700000000000006,"5min":0.77400000000000002,"15min":0.78800000000000003},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.877,"5min":1.2549999999999999,"15min":1.2549999999999999},"last":0.93999999999999995},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.70199999999999996,"15min":0.70999999999999996},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.79700000000000004,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.78300000000000003}]},{"osd":11,"last update":"Fri Jul 14 19:02:53 2023","interfaces":[{"interface":"back","average":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":1.01,"5min":1.01,"15min":1.01},"last":0.754},{"interface":"front","average":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":1.052,"5min":1.052,"15min":1.052},"last":0.91400000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28044,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717108,"statfs":{"total":95995035648,"available":95966318592,"internally_reserved":0,"allocated":1048576,"data_stored":530628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.38100000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.78700000000000003,"5min":1.111,"15min":1.111},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51600000000000001,"15min":0.51400000000000001},"min":{"1min":0.35199999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.61399999999999999}]},{"osd":1,"last update":"Fri Jul 14 19:02:43 2023","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.54800000000000004,"15min":0.52800000000000002},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.628,"5min":0.55200000000000005,"15min":0.52400000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.80600000000000005}]},{"osd":3,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53700000000000003,"15min":0.53500000000000003},"min":{"1min":0.32800000000000001,"5min":0.308,"15min":0.308},"max":{"1min":0.77000000000000002,"5min":0.84199999999999997,"15min":0.84199999999999997},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.55200000000000005,"15min":0.53900000000000003},"min":{"1min":0.45500000000000002,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.78100000000000003,"5min":1.046,"15min":1.046},"last":0.74199999999999999}]},{"osd":4,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.66700000000000004,"15min":0.66700000000000004},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.78600000000000003}]},{"osd":5,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.44500000000000001,"15min":0.41799999999999998},"min":{"1min":0.33600000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.2190000000000001,"5min":1.2190000000000001,"15min":1.2190000000000001},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.505,"15min":0.47299999999999998},"min":{"1min":0.36799999999999999,"5min":0.308,"15min":0.308},"max":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.88800000000000001}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.59599999999999997,"15min":0.56599999999999995},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.89600000000000002,"5min":0.91900000000000004,"15min":0.91900000000000004},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60299999999999998,"15min":0.58799999999999997},"min":{"1min":0.42199999999999999,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.89300000000000002,"5min":0.96199999999999997,"15min":0.96199999999999997},"last":0.70199999999999996}]},{"osd":7,"last update":"Fri Jul 14 19:03:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"last":0.98299999999999998},{"interface":"front","average":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"last":0.77100000000000002}]},{"osd":8,"last update":"Fri Jul 14 19:02:51 2023","interfaces":[{"interface":"back","average":{"1min":0.66900000000000004,"5min":0.56499999999999995,"15min":0.54700000000000004},"min":{"1min":0.443,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.83599999999999997},{"interface":"front","average":{"1min":0.68899999999999995,"5min":0.59599999999999997,"15min":0.57999999999999996},"min":{"1min":0.45500000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.95399999999999996,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.94499999999999995}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.68899999999999995,"15min":0.68300000000000005},"min":{"1min":0.45800000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.92600000000000005,"5min":1.0940000000000001,"15min":1.0940000000000001},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.77900000000000003,"5min":0.72799999999999998,"15min":0.71999999999999997},"min":{"1min":0.58799999999999997,"5min":0.52500000000000002,"15min":0.52500000000000002},"max":{"1min":1.091,"5min":1.091,"15min":1.091},"last":0.875}]},{"osd":10,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.88500000000000001,"5min":0.753,"15min":0.73099999999999998},"min":{"1min":0.69799999999999995,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.1259999999999999,"5min":1.1259999999999999,"15min":1.1259999999999999},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.90200000000000002,"5min":0.752,"15min":0.72699999999999998},"min":{"1min":0.68700000000000006,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.1080000000000001,"5min":1.1080000000000001,"15min":1.1080000000000001},"last":1.0349999999999999}]},{"osd":11,"last update":"Fri Jul 14 19:03:31 2023","interfaces":[{"interface":"back","average":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.1599999999999999,"5min":1.1599999999999999,"15min":1.1599999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.1379999999999999,"5min":1.1379999999999999,"15min":1.1379999999999999},"last":1.0109999999999999}]}]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":472284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34200000000000003,"15min":0.33000000000000002},"min":{"1min":0.26900000000000002,"5min":0.188,"15min":0.188},"max":{"1min":0.57499999999999996,"5min":0.57499999999999996,"15min":0.57499999999999996},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.35299999999999998,"15min":0.34300000000000003},"min":{"1min":0.251,"5min":0.215,"15min":0.215},"max":{"1min":0.52800000000000002,"5min":0.55400000000000005,"15min":0.55400000000000005},"last":0.28599999999999998}]},{"osd":1,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51200000000000001,"15min":0.503},"min":{"1min":0.32900000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.71799999999999997,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.85699999999999998,"5min":1.1919999999999999,"15min":1.1919999999999999},"last":0.496}]},{"osd":2,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.62,"15min":0.65700000000000003},"min":{"1min":0.39600000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.745,"5min":4.6509999999999998,"15min":4.6509999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.60799999999999998,"15min":0.66400000000000003},"min":{"1min":0.36599999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.79500000000000004,"5min":4.6340000000000003,"15min":4.6340000000000003},"last":0.47099999999999997}]},{"osd":4,"last update":"Fri Jul 14 19:03:27 2023","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.80600000000000005,"5min":0.93999999999999995,"15min":0.93999999999999995},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.57599999999999996}]},{"osd":5,"last update":"Fri Jul 14 19:02:44 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.58699999999999997,"15min":0.57799999999999996},"min":{"1min":0.46200000000000002,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.78200000000000003,"5min":1.0640000000000001,"15min":1.0640000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.61299999999999999,"15min":0.60699999999999998},"min":{"1min":0.498,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"last":0.54900000000000004}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.48299999999999998},"min":{"1min":0.34499999999999997,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.74399999999999999,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54200000000000004,"15min":0.53800000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.82799999999999996,"5min":1.119,"15min":1.119},"last":0.433}]},{"osd":7,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.59099999999999997,"15min":0.58799999999999997},"min":{"1min":0.42599999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.89600000000000002},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.60499999999999998,"15min":0.59799999999999998},"min":{"1min":0.376,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.48499999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:01:47 2023","interfaces":[{"interface":"back","average":{"1min":0.63500000000000001,"5min":0.63500000000000001,"15min":0.63500000000000001},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.499},{"interface":"front","average":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.54600000000000004}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.57099999999999995,"15min":0.58199999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.66200000000000003,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.57799999999999996,"5min":0.63900000000000001,"15min":0.64900000000000002},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.76000000000000001,"5min":0.92400000000000004,"15min":0.92400000000000004},"last":0.53600000000000003}]},{"osd":10,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.68899999999999995,"15min":0.69699999999999995},"min":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.91900000000000004,"5min":0.98099999999999998,"15min":0.98099999999999998},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.66600000000000004,"5min":0.69799999999999995,"15min":0.70399999999999996},"min":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"max":{"1min":0.93300000000000005,"5min":0.93300000000000005,"15min":0.93300000000000005},"last":0.56699999999999995}]},{"osd":11,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":0.84099999999999997,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.64300000000000002,"15min":0.64300000000000002},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.60099999999999998}]}]},{"osd":4,"up_from":33,"seq":141733920825,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":974848,"data_stored":476809,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:28 2023","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.498,"15min":0.46600000000000003},"min":{"1min":0.39800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.0760000000000001,"5min":1.0760000000000001,"15min":1.0760000000000001},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.503,"15min":0.47699999999999998},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.38100000000000001}]},{"osd":1,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.38400000000000001,"15min":0.34799999999999998},"min":{"1min":0.33600000000000002,"5min":0.222,"15min":0.222},"max":{"1min":0.59199999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.40100000000000002,"15min":0.36499999999999999},"min":{"1min":0.34399999999999997,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.71799999999999997,"5min":0.83299999999999996,"15min":0.83299999999999996},"last":0.55500000000000005}]},{"osd":2,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.64200000000000002,"5min":0.56999999999999995,"15min":0.53900000000000003},"min":{"1min":0.39600000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.89500000000000002,"5min":0.95199999999999996,"15min":0.95199999999999996},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.58499999999999996,"15min":0.54300000000000004},"min":{"1min":0.45500000000000002,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.80200000000000005,"5min":1.105,"15min":1.105},"last":0.82099999999999995}]},{"osd":3,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.55300000000000005,"15min":0.51200000000000001},"min":{"1min":0.432,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.72299999999999998,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.54900000000000004,"15min":0.50600000000000001},"min":{"1min":0.40899999999999997,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.81899999999999995,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.58699999999999997}]},{"osd":5,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.64500000000000002,"5min":0.57599999999999996,"15min":0.55200000000000005},"min":{"1min":0.36499999999999999,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.77600000000000002},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.57499999999999996,"15min":0.54300000000000004},"min":{"1min":0.49099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.127,"5min":1.127,"15min":1.127},"last":0.72399999999999998}]},{"osd":6,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.627,"5min":0.57899999999999996,"15min":0.57099999999999995},"min":{"1min":0.41099999999999998,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.83899999999999997,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.61799999999999999,"15min":0.60399999999999998},"min":{"1min":0.45600000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.93799999999999994,"5min":1.038,"15min":1.038},"last":0.64500000000000002}]},{"osd":7,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.57999999999999996,"15min":0.55600000000000005},"min":{"1min":0.442,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.46200000000000002,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.873,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.86099999999999999}]},{"osd":8,"last update":"Fri Jul 14 19:02:49 2023","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.68600000000000005,"15min":0.68100000000000005},"min":{"1min":0.54200000000000004,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":1.0609999999999999},"last":0.89200000000000002},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.1479999999999999,"5min":1.1479999999999999,"15min":1.1479999999999999},"last":0.90500000000000003}]},{"osd":9,"last update":"Fri Jul 14 19:03:09 2023","interfaces":[{"interface":"back","average":{"1min":0.70499999999999996,"5min":0.66900000000000004,"15min":0.66300000000000003},"min":{"1min":0.46200000000000002,"5min":0.46200000000000002,"15min":0.46200000000000002},"max":{"1min":0.997,"5min":0.997,"15min":0.997},"last":0.746},{"interface":"front","average":{"1min":0.73099999999999998,"5min":0.72299999999999998,"15min":0.72199999999999998},"min":{"1min":0.58399999999999996,"5min":0.51000000000000001,"15min":0.51000000000000001},"max":{"1min":1.008,"5min":1.008,"15min":1.008},"last":0.874}]},{"osd":10,"last update":"Fri Jul 14 19:03:38 2023","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.68700000000000006,"15min":0.68500000000000005},"min":{"1min":0.53100000000000003,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.92500000000000004},"last":0.92000000000000004},{"interface":"front","average":{"1min":0.75700000000000001,"5min":0.71799999999999997,"15min":0.71199999999999997},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.93300000000000005}]},{"osd":11,"last update":"Fri Jul 14 19:03:30 2023","interfaces":[{"interface":"back","average":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":0.98199999999999998,"5min":0.98199999999999998,"15min":0.98199999999999998},"last":0.94999999999999996},{"interface":"front","average":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.0329999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"last":0.96999999999999997}]}]},{"osd":5,"up_from":38,"seq":163208757300,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28572,"kb_used_data":1548,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716580,"statfs":{"total":95995035648,"available":95965777920,"internally_reserved":0,"allocated":1585152,"data_stored":1067951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.53800000000000003,"15min":0.52100000000000002},"min":{"1min":0.42399999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53300000000000003,"15min":0.51900000000000002},"min":{"1min":0.34300000000000003,"5min":0.312,"15min":0.312},"max":{"1min":0.86899999999999999,"5min":0.86899999999999999,"15min":0.86899999999999999},"last":0.57299999999999995}]},{"osd":1,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.52300000000000002,"15min":0.501},"min":{"1min":0.41899999999999998,"5min":0.373,"15min":0.373},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.66700000000000004,"5min":0.56000000000000005,"15min":0.53200000000000003},"min":{"1min":0.39200000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.69299999999999995}]},{"osd":2,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.39100000000000001,"15min":0.36299999999999999},"min":{"1min":0.317,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.996,"5min":0.996,"15min":0.996},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.41399999999999998,"15min":0.38900000000000001},"min":{"1min":0.309,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":1.1619999999999999,"5min":1.1619999999999999,"15min":1.1619999999999999},"last":0.44600000000000001}]},{"osd":3,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55800000000000005,"15min":0.54700000000000004},"min":{"1min":0.36499999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70599999999999996,"5min":0.85999999999999999,"15min":0.85999999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55700000000000005,"15min":0.54700000000000004},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.75900000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.63100000000000001}]},{"osd":4,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.65200000000000002,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.42599999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.0920000000000001,"5min":1.0920000000000001,"15min":1.0920000000000001},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.58799999999999997,"15min":0.57099999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.91100000000000003,"5min":0.91100000000000003,"15min":0.91100000000000003},"last":0.67700000000000005}]},{"osd":6,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.60199999999999998,"15min":0.58799999999999997},"min":{"1min":0.36599999999999999,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.84099999999999997,"5min":0.95399999999999996,"15min":0.95399999999999996},"last":0.751},{"interface":"front","average":{"1min":0.66900000000000004,"5min":0.64500000000000002,"15min":0.63100000000000001},"min":{"1min":0.40200000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":1.044,"5min":1.044,"15min":1.044},"last":0.78700000000000003}]},{"osd":7,"last update":"Fri Jul 14 19:03:28 2023","interfaces":[{"interface":"back","average":{"1min":0.71299999999999997,"5min":0.63900000000000001,"15min":0.622},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.97499999999999998,"5min":0.97499999999999998,"15min":0.97499999999999998},"last":0.72399999999999998},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.65100000000000002,"15min":0.64000000000000001},"min":{"1min":0.46000000000000002,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.89000000000000001},"last":0.80800000000000005}]},{"osd":8,"last update":"Fri Jul 14 19:02:46 2023","interfaces":[{"interface":"back","average":{"1min":0.66200000000000003,"5min":0.63800000000000001,"15min":0.63400000000000001},"min":{"1min":0.41599999999999998,"5min":0.374,"15min":0.374},"max":{"1min":1.1279999999999999,"5min":1.1279999999999999,"15min":1.1279999999999999},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.64300000000000002,"15min":0.64900000000000002},"min":{"1min":0.40200000000000002,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.03,"5min":1.03,"15min":1.03},"last":0.77600000000000002}]},{"osd":9,"last update":"Fri Jul 14 19:03:08 2023","interfaces":[{"interface":"back","average":{"1min":0.72799999999999998,"5min":0.68700000000000006,"15min":0.68000000000000005},"min":{"1min":0.49299999999999999,"5min":0.432,"15min":0.432},"max":{"1min":1.258,"5min":1.258,"15min":1.258},"last":0.76300000000000001},{"interface":"front","average":{"1min":0.76500000000000001,"5min":0.755,"15min":0.753},"min":{"1min":0.51900000000000002,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.1950000000000001,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.86899999999999999}]},{"osd":10,"last update":"Fri Jul 14 19:03:33 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.71399999999999997,"15min":0.69699999999999995},"min":{"1min":0.67500000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.79000000000000004,"5min":0.71099999999999997,"15min":0.69799999999999995},"min":{"1min":0.623,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":1.0209999999999999,"5min":1.0209999999999999,"15min":1.0209999999999999},"last":0.85399999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":56970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":58628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":58890,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-07-14T19:03:48.855 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-07-14T19:03:48.855 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-07-14T19:03:48.856 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-07-14T19:03:48.856 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph health --format=json 2023-07-14T19:03:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:49.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:49 smithi088 ceph-mon[128973]: pgmap v296: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:49.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:49 smithi088 ceph-mon[128973]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:49.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:49 smithi078 ceph-mon[127425]: pgmap v296: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:49.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:49 smithi078 ceph-mon[127425]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:49.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:49 smithi102 ceph-mon[129280]: pgmap v296: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:49.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:49 smithi102 ceph-mon[129280]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-07-14T19:03:50.681 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.677+0000 7f939f477700 1 -- 172.21.15.78:0/654315280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980fe4b0 msgr2=0x7f93980fe880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:50.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.677+0000 7f939f477700 1 --2- 172.21.15.78:0/654315280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980fe4b0 0x7f93980fe880 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f93880099c0 tx=0x7f9388009cd0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 -- 172.21.15.78:0/654315280 shutdown_connections 2023-07-14T19:03:50.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 --2- 172.21.15.78:0/654315280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9398105010 0x7f939810cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 --2- 172.21.15.78:0/654315280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980fe4b0 0x7f93980fe880 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 --2- 172.21.15.78:0/654315280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93980fee50 0x7f9398104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.682 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 -- 172.21.15.78:0/654315280 >> 172.21.15.78:0/654315280 conn(0x7f9398075280 msgr2=0x7f9398075680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 -- 172.21.15.78:0/654315280 shutdown_connections 2023-07-14T19:03:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 -- 172.21.15.78:0/654315280 wait complete. 2023-07-14T19:03:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 Processor -- start 2023-07-14T19:03:50.683 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 -- start start 2023-07-14T19:03:50.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93980fe4b0 0x7f939819ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:50.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93980fee50 0x7f939819bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:50.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9398105010 0x7f9398195aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:50.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9398196110 con 0x7f9398105010 2023-07-14T19:03:50.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9398196280 con 0x7f93980fe4b0 2023-07-14T19:03:50.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.678+0000 7f939f477700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9398196440 con 0x7f93980fee50 2023-07-14T19:03:50.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9398105010 0x7f9398195aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:50.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9398105010 0x7f9398195aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35524/0 (socket says 172.21.15.78:35524) 2023-07-14T19:03:50.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 -- 172.21.15.78:0/2269107481 learned_addr learned my addr 172.21.15.78:0/2269107481 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:50.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 -- 172.21.15.78:0/2269107481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93980fee50 msgr2=0x7f939819bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:50.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93980fee50 0x7f939819bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939d213700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93980fe4b0 0x7f939819ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:50.686 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 -- 172.21.15.78:0/2269107481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93980fe4b0 msgr2=0x7f939819ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:50.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93980fe4b0 0x7f939819ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 -- 172.21.15.78:0/2269107481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93880095f0 con 0x7f9398105010 2023-07-14T19:03:50.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939da14700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9398105010 0x7f9398195aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9390007ee0 tx=0x7f939000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:50.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f938e7fc700 1 -- 172.21.15.78:0/2269107481 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939000ddd0 con 0x7f9398105010 2023-07-14T19:03:50.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f938e7fc700 1 -- 172.21.15.78:0/2269107481 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f939000f040 con 0x7f9398105010 2023-07-14T19:03:50.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9398196720 con 0x7f9398105010 2023-07-14T19:03:50.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f938e7fc700 1 -- 172.21.15.78:0/2269107481 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9390004870 con 0x7f9398105010 2023-07-14T19:03:50.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.679+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93981a2a10 con 0x7f9398105010 2023-07-14T19:03:50.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.680+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939804e340 con 0x7f9398105010 2023-07-14T19:03:50.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.681+0000 7f938e7fc700 1 -- 172.21.15.78:0/2269107481 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9390003a00 con 0x7f9398105010 2023-07-14T19:03:50.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.682+0000 7f938e7fc700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93840a11e0 0x7f93840a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:50.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.682+0000 7f939d213700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93840a11e0 0x7f93840a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:50.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.682+0000 7f938e7fc700 1 -- 172.21.15.78:0/2269107481 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f93900ccfe0 con 0x7f9398105010 2023-07-14T19:03:50.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.683+0000 7f939d213700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93840a11e0 0x7f93840a3690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9388006010 tx=0x7f9388000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:50.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.686+0000 7f938e7fc700 1 -- 172.21.15.78:0/2269107481 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f93900962c0 con 0x7f9398105010 2023-07-14T19:03:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.910+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9398196fb0 con 0x7f9398105010 2023-07-14T19:03:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.910+0000 7f938e7fc700 1 -- 172.21.15.78:0/2269107481 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f93900998e0 con 0x7f9398105010 2023-07-14T19:03:50.915 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:50.915 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-07-14T19:03:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93840a11e0 msgr2=0x7f93840a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93840a11e0 0x7f93840a3690 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f9388006010 tx=0x7f9388000bc0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9398105010 msgr2=0x7f9398195aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9398105010 0x7f9398195aa0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f9390007ee0 tx=0x7f939000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 shutdown_connections 2023-07-14T19:03:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93840a11e0 0x7f93840a3690 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93980fe4b0 0x7f939819ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9398105010 0x7f9398195aa0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 --2- 172.21.15.78:0/2269107481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93980fee50 0x7f939819bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.912+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 >> 172.21.15.78:0/2269107481 conn(0x7f9398075280 msgr2=0x7f9398103910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.913+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 shutdown_connections 2023-07-14T19:03:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:50.913+0000 7f939f477700 1 -- 172.21.15.78:0/2269107481 wait complete. 2023-07-14T19:03:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:51 smithi078 ceph-mon[127425]: pgmap v297: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:03:51.291 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:51 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2269107481' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:03:51.352 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-07-14T19:03:51.352 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:03:51.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:51 smithi088 ceph-mon[128973]: pgmap v297: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:03:51.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:51 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2269107481' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:03:51.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 -- 172.21.15.78:0/1047369594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48040ff020 msgr2=0x7f48040ff430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:51.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1047369594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48040ff020 0x7f48040ff430 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009990 tx=0x7f47f4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 -- 172.21.15.78:0/1047369594 shutdown_connections 2023-07-14T19:03:51.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1047369594 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48041006e0 0x7f4804106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1047369594 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48040ff020 0x7f48040ff430 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1047369594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48040ffd30 0x7f48041001a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 -- 172.21.15.78:0/1047369594 >> 172.21.15.78:0/1047369594 conn(0x7f48040f98a0 msgr2=0x7f48040fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:51.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 -- 172.21.15.78:0/1047369594 shutdown_connections 2023-07-14T19:03:51.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.473+0000 7f480ba1d700 1 -- 172.21.15.78:0/1047369594 wait complete. 2023-07-14T19:03:51.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f480ba1d700 1 Processor -- start 2023-07-14T19:03:51.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f480ba1d700 1 -- start start 2023-07-14T19:03:51.480 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f480ba1d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48040ff020 0x7f48040fe6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:51.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f480ba1d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48040ffd30 0x7f480419a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:51.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f480ba1d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48041006e0 0x7f480419cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:51.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f480ba1d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480419d410 con 0x7f48041006e0 2023-07-14T19:03:51.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f480ba1d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480419d580 con 0x7f48040ffd30 2023-07-14T19:03:51.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f480ba1d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f480419d740 con 0x7f48040ff020 2023-07-14T19:03:51.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48041006e0 0x7f480419cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:51.481 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48041006e0 0x7f480419cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35536/0 (socket says 172.21.15.78:35536) 2023-07-14T19:03:51.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 -- 172.21.15.78:0/1666545714 learned_addr learned my addr 172.21.15.78:0/1666545714 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:51.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4808fb8700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48040ffd30 0x7f480419a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:51.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f48097b9700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48040ff020 0x7f48040fe6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:51.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 -- 172.21.15.78:0/1666545714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48040ff020 msgr2=0x7f48040fe6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:51.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48040ff020 0x7f48040fe6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 -- 172.21.15.78:0/1666545714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48040ffd30 msgr2=0x7f480419a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:51.482 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48040ffd30 0x7f480419a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 -- 172.21.15.78:0/1666545714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f40095f0 con 0x7f48041006e0 2023-07-14T19:03:51.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f48097b9700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48040ff020 0x7f48040fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:51.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.474+0000 7f4809fba700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48041006e0 0x7f480419cc70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f47fc007ee0 tx=0x7f47fc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:51.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.475+0000 7f47fa7fc700 1 -- 172.21.15.78:0/1666545714 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc017070 con 0x7f48041006e0 2023-07-14T19:03:51.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.475+0000 7f47fa7fc700 1 -- 172.21.15.78:0/1666545714 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f47fc00f040 con 0x7f48041006e0 2023-07-14T19:03:51.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.475+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f480419da20 con 0x7f48041006e0 2023-07-14T19:03:51.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.475+0000 7f47fa7fc700 1 -- 172.21.15.78:0/1666545714 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47fc005710 con 0x7f48041006e0 2023-07-14T19:03:51.483 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.475+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f480419df60 con 0x7f48041006e0 2023-07-14T19:03:51.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.476+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4804105a70 con 0x7f48041006e0 2023-07-14T19:03:51.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.477+0000 7f47fa7fc700 1 -- 172.21.15.78:0/1666545714 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f47fc005870 con 0x7f48041006e0 2023-07-14T19:03:51.484 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.477+0000 7f47fa7fc700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f47f00a12a0 0x7f47f00a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:51.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.477+0000 7f48097b9700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f47f00a12a0 0x7f47f00a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:51.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.477+0000 7f47fa7fc700 1 -- 172.21.15.78:0/1666545714 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f47fc0ccc20 con 0x7f48041006e0 2023-07-14T19:03:51.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.478+0000 7f48097b9700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f47f00a12a0 0x7f47f00a3750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f47f40095c0 tx=0x7f47f4009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:51.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.481+0000 7f47fa7fc700 1 -- 172.21.15.78:0/1666545714 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f47fc095ea0 con 0x7f48041006e0 2023-07-14T19:03:51.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:51 smithi102 ceph-mon[129280]: pgmap v297: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:03:51.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:51 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2269107481' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-07-14T19:03:51.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.670+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f480402cab0 con 0x7f48041006e0 2023-07-14T19:03:51.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.672+0000 7f47fa7fc700 1 -- 172.21.15.78:0/1666545714 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24453 (secure 0 0 0) 0x7f47fc0994c0 con 0x7f48041006e0 2023-07-14T19:03:51.676 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:51.676 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:03:31.201384+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:03:51.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f47f00a12a0 msgr2=0x7f47f00a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:51.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f47f00a12a0 0x7f47f00a3750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f47f40095c0 tx=0x7f47f4009510 comp rx=0 tx=0).stop 2023-07-14T19:03:51.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48041006e0 msgr2=0x7f480419cc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:51.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48041006e0 0x7f480419cc70 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f47fc007ee0 tx=0x7f47fc00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 shutdown_connections 2023-07-14T19:03:51.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f47f00a12a0 0x7f47f00a3750 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f48041006e0 0x7f480419cc70 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f48040ff020 0x7f48040fe6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 --2- 172.21.15.78:0/1666545714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f48040ffd30 0x7f480419a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 >> 172.21.15.78:0/1666545714 conn(0x7f48040f98a0 msgr2=0x7f48040fbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:51.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 shutdown_connections 2023-07-14T19:03:51.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.674+0000 7f480ba1d700 1 -- 172.21.15.78:0/1666545714 wait complete. 2023-07-14T19:03:51.691 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:02:28.269050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:02:29.547771+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T19:02:32.797605+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T19:03:51.692 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:03:51.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 -- 172.21.15.78:0/377475978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 msgr2=0x7fef0c106fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:51.818 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 --2- 172.21.15.78:0/377475978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 0x7fef0c106fc0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fef0400b0a0 tx=0x7fef0400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 -- 172.21.15.78:0/377475978 shutdown_connections 2023-07-14T19:03:51.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 --2- 172.21.15.78:0/377475978 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0c0feb80 0x7fef0c0fef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 --2- 172.21.15.78:0/377475978 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 0x7fef0c106fc0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 --2- 172.21.15.78:0/377475978 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef0c0ffd70 0x7fef0c1001c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 -- 172.21.15.78:0/377475978 >> 172.21.15.78:0/377475978 conn(0x7fef0c0fa1a0 msgr2=0x7fef0c0fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:51.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 -- 172.21.15.78:0/377475978 shutdown_connections 2023-07-14T19:03:51.819 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.814+0000 7fef12167700 1 -- 172.21.15.78:0/377475978 wait complete. 2023-07-14T19:03:51.820 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef12167700 1 Processor -- start 2023-07-14T19:03:51.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef12167700 1 -- start start 2023-07-14T19:03:51.821 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef12167700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef0c0feb80 0x7fef0c102b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef12167700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0c0ffd70 0x7fef0c101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef12167700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 0x7fef0c101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef12167700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c101e30 con 0x7fef0c100700 2023-07-14T19:03:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef12167700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c101fa0 con 0x7fef0c0ffd70 2023-07-14T19:03:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef12167700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c19f980 con 0x7fef0c0feb80 2023-07-14T19:03:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef0bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 0x7fef0c101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:51.822 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef0bfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 0x7fef0c101690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35550/0 (socket says 172.21.15.78:35550) 2023-07-14T19:03:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef0bfff700 1 -- 172.21.15.78:0/3052695373 learned_addr learned my addr 172.21.15.78:0/3052695373 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.815+0000 7fef0b7fe700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef0c0feb80 0x7fef0c102b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef0affd700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0c0ffd70 0x7fef0c101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef0bfff700 1 -- 172.21.15.78:0/3052695373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef0c0feb80 msgr2=0x7fef0c102b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef0bfff700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef0c0feb80 0x7fef0c102b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef0bfff700 1 -- 172.21.15.78:0/3052695373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0c0ffd70 msgr2=0x7fef0c101150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef0bfff700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0c0ffd70 0x7fef0c101150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:51.823 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef0bfff700 1 -- 172.21.15.78:0/3052695373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef04009d00 con 0x7fef0c100700 2023-07-14T19:03:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef0affd700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0c0ffd70 0x7fef0c101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef0bfff700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 0x7fef0c101690 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fef0400b070 tx=0x7fef04004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef08ff9700 1 -- 172.21.15.78:0/3052695373 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef0400e040 con 0x7fef0c100700 2023-07-14T19:03:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef08ff9700 1 -- 172.21.15.78:0/3052695373 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fef04003e90 con 0x7fef0c100700 2023-07-14T19:03:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef08ff9700 1 -- 172.21.15.78:0/3052695373 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef04004020 con 0x7fef0c100700 2023-07-14T19:03:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0c19fc00 con 0x7fef0c100700 2023-07-14T19:03:51.824 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.816+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0c1a0030 con 0x7fef0c100700 2023-07-14T19:03:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.817+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0c10a1f0 con 0x7fef0c100700 2023-07-14T19:03:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.818+0000 7fef08ff9700 1 -- 172.21.15.78:0/3052695373 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fef04003a00 con 0x7fef0c100700 2023-07-14T19:03:51.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.818+0000 7fef08ff9700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7feef40a11e0 0x7feef40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:51.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.819+0000 7fef0b7fe700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7feef40a11e0 0x7feef40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:51.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.819+0000 7fef08ff9700 1 -- 172.21.15.78:0/3052695373 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fef040ce4e0 con 0x7fef0c100700 2023-07-14T19:03:51.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.819+0000 7fef0b7fe700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7feef40a11e0 0x7feef40a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0ffbd0 tx=0x7feefc00a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:51.827 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:51.823+0000 7fef08ff9700 1 -- 172.21.15.78:0/3052695373 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fef040977f0 con 0x7fef0c100700 2023-07-14T19:03:52.013 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.010+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fef0c1a05e0 con 0x7fef0c100700 2023-07-14T19:03:52.014 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.010+0000 7fef08ff9700 1 -- 172.21.15.78:0/3052695373 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fef0409ae10 con 0x7fef0c100700 2023-07-14T19:03:52.014 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:03:52.015 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7feef40a11e0 msgr2=0x7feef40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7feef40a11e0 0x7feef40a3690 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fef0c0ffbd0 tx=0x7feefc00a3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 msgr2=0x7fef0c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 0x7fef0c101690 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fef0400b070 tx=0x7fef04004c80 comp rx=0 tx=0).stop 2023-07-14T19:03:52.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 shutdown_connections 2023-07-14T19:03:52.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7feef40a11e0 0x7feef40a3690 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.016 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fef0c100700 0x7fef0c101690 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fef0c0feb80 0x7fef0c102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 --2- 172.21.15.78:0/3052695373 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fef0c0ffd70 0x7fef0c101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 >> 172.21.15.78:0/3052695373 conn(0x7fef0c0fa1a0 msgr2=0x7fef0c0689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:52.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 shutdown_connections 2023-07-14T19:03:52.017 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.012+0000 7fef12167700 1 -- 172.21.15.78:0/3052695373 wait complete. 2023-07-14T19:03:52.028 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:03:52.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 -- 172.21.15.78:0/778270822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00ffd00 msgr2=0x7f42d0100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 --2- 172.21.15.78:0/778270822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00ffd00 0x7f42d0100150 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009990 tx=0x7f42cc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 -- 172.21.15.78:0/778270822 shutdown_connections 2023-07-14T19:03:52.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 --2- 172.21.15.78:0/778270822 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d0100690 0x7f42d0106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.152 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 --2- 172.21.15.78:0/778270822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42d00feb10 0x7f42d00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 --2- 172.21.15.78:0/778270822 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00ffd00 0x7f42d0100150 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 -- 172.21.15.78:0/778270822 >> 172.21.15.78:0/778270822 conn(0x7f42d00fa130 msgr2=0x7f42d00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:52.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 -- 172.21.15.78:0/778270822 shutdown_connections 2023-07-14T19:03:52.153 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.148+0000 7f42d790c700 1 -- 172.21.15.78:0/778270822 wait complete. 2023-07-14T19:03:52.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d790c700 1 Processor -- start 2023-07-14T19:03:52.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d790c700 1 -- start start 2023-07-14T19:03:52.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d790c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00feb10 0x7f42d0068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d790c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d00ffd00 0x7f42d019a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d790c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42d0100690 0x7f42d019cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.155 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d790c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d019d420 con 0x7f42d00feb10 2023-07-14T19:03:52.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d790c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d019d590 con 0x7f42d00ffd00 2023-07-14T19:03:52.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d790c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d019d750 con 0x7f42d0100690 2023-07-14T19:03:52.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00feb10 0x7f42d0068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00feb10 0x7f42d0068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35560/0 (socket says 172.21.15.78:35560) 2023-07-14T19:03:52.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 -- 172.21.15.78:0/1361137081 learned_addr learned my addr 172.21.15.78:0/1361137081 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:52.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d4ea7700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d00ffd00 0x7f42d019a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.156 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d5ea9700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42d0100690 0x7f42d019cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 -- 172.21.15.78:0/1361137081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42d0100690 msgr2=0x7f42d019cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42d0100690 0x7f42d019cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 -- 172.21.15.78:0/1361137081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d00ffd00 msgr2=0x7f42d019a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d00ffd00 0x7f42d019a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 -- 172.21.15.78:0/1361137081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42cc0095f0 con 0x7f42d00feb10 2023-07-14T19:03:52.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42d56a8700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00feb10 0x7f42d0068330 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f42c400c970 tx=0x7f42c400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:52.157 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.149+0000 7f42c27fc700 1 -- 172.21.15.78:0/1361137081 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4004220 con 0x7f42d00feb10 2023-07-14T19:03:52.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.150+0000 7f42c27fc700 1 -- 172.21.15.78:0/1361137081 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f42c4007d30 con 0x7f42d00feb10 2023-07-14T19:03:52.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.150+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42d019da30 con 0x7f42d00feb10 2023-07-14T19:03:52.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.150+0000 7f42c27fc700 1 -- 172.21.15.78:0/1361137081 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42c4003b80 con 0x7f42d00feb10 2023-07-14T19:03:52.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.150+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42d019df10 con 0x7f42d00feb10 2023-07-14T19:03:52.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.150+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42d010a190 con 0x7f42d00feb10 2023-07-14T19:03:52.158 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.152+0000 7f42c27fc700 1 -- 172.21.15.78:0/1361137081 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f42c4003ce0 con 0x7f42d00feb10 2023-07-14T19:03:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.152+0000 7f42c27fc700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42bc0a11e0 0x7f42bc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.152+0000 7f42c27fc700 1 -- 172.21.15.78:0/1361137081 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f42c40ccd80 con 0x7f42d00feb10 2023-07-14T19:03:52.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.152+0000 7f42d4ea7700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42bc0a11e0 0x7f42bc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.153+0000 7f42d4ea7700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42bc0a11e0 0x7f42bc0a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00c010 tx=0x7f42cc005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:52.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.155+0000 7f42c27fc700 1 -- 172.21.15.78:0/1361137081 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f42c4096060 con 0x7f42d00feb10 2023-07-14T19:03:52.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:52 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1666545714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:52.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:52 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3052695373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:03:52.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.344+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f42d0105ac0 con 0x7f42d00feb10 2023-07-14T19:03:52.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.344+0000 7f42c27fc700 1 -- 172.21.15.78:0/1361137081 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f42c4099680 con 0x7f42d00feb10 2023-07-14T19:03:52.349 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:03:52.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42bc0a11e0 msgr2=0x7f42bc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42bc0a11e0 0x7f42bc0a3690 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f42cc00c010 tx=0x7f42cc005960 comp rx=0 tx=0).stop 2023-07-14T19:03:52.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00feb10 msgr2=0x7f42d0068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00feb10 0x7f42d0068330 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f42c400c970 tx=0x7f42c400cc80 comp rx=0 tx=0).stop 2023-07-14T19:03:52.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 shutdown_connections 2023-07-14T19:03:52.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42bc0a11e0 0x7f42bc0a3690 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42d0100690 0x7f42d019cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f42d00feb10 0x7f42d0068330 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 --2- 172.21.15.78:0/1361137081 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42d00ffd00 0x7f42d019a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 >> 172.21.15.78:0/1361137081 conn(0x7f42d00fa130 msgr2=0x7f42d00fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:52.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.346+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 shutdown_connections 2023-07-14T19:03:52.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.347+0000 7f42d790c700 1 -- 172.21.15.78:0/1361137081 wait complete. 2023-07-14T19:03:52.363 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:03:52.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:52 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1666545714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:52.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:52 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3052695373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:03:52.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 -- 172.21.15.78:0/1517021412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3510100690 msgr2=0x7f3510106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 --2- 172.21.15.78:0/1517021412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3510100690 0x7f3510106f50 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3508009990 tx=0x7f3508009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 -- 172.21.15.78:0/1517021412 shutdown_connections 2023-07-14T19:03:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 --2- 172.21.15.78:0/1517021412 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3510100690 0x7f3510106f50 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 --2- 172.21.15.78:0/1517021412 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35100feb10 0x7f35100fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 --2- 172.21.15.78:0/1517021412 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f35100ffd00 0x7f3510100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 -- 172.21.15.78:0/1517021412 >> 172.21.15.78:0/1517021412 conn(0x7f35100fa130 msgr2=0x7f35100fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:52.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 -- 172.21.15.78:0/1517021412 shutdown_connections 2023-07-14T19:03:52.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.483+0000 7f3517805700 1 -- 172.21.15.78:0/1517021412 wait complete. 2023-07-14T19:03:52.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3517805700 1 Processor -- start 2023-07-14T19:03:52.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3517805700 1 -- start start 2023-07-14T19:03:52.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3517805700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35100feb10 0x7f3510068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3517805700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35100ffd00 0x7f351019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3517805700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3510100690 0x7f351019cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3517805700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351019d380 con 0x7f35100feb10 2023-07-14T19:03:52.491 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3517805700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351019d4f0 con 0x7f3510100690 2023-07-14T19:03:52.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3517805700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f351019d6b0 con 0x7f35100ffd00 2023-07-14T19:03:52.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f35155a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35100feb10 0x7f3510068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f35155a1700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35100feb10 0x7f3510068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35574/0 (socket says 172.21.15.78:35574) 2023-07-14T19:03:52.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f35155a1700 1 -- 172.21.15.78:0/3371001201 learned_addr learned my addr 172.21.15.78:0/3371001201 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:52.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3514da0700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35100ffd00 0x7f351019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f3515da2700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3510100690 0x7f351019cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.492 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f35155a1700 1 -- 172.21.15.78:0/3371001201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35100ffd00 msgr2=0x7f351019a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f35155a1700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35100ffd00 0x7f351019a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f35155a1700 1 -- 172.21.15.78:0/3371001201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3510100690 msgr2=0x7f351019cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f35155a1700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3510100690 0x7f351019cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.484+0000 7f35155a1700 1 -- 172.21.15.78:0/3371001201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35080095f0 con 0x7f35100feb10 2023-07-14T19:03:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.485+0000 7f3514da0700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35100ffd00 0x7f351019a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.485+0000 7f3515da2700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3510100690 0x7f351019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:52.493 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.485+0000 7f35155a1700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35100feb10 0x7f3510068330 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f350000efc0 tx=0x7f350000c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.485+0000 7f350e7fc700 1 -- 172.21.15.78:0/3371001201 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500018070 con 0x7f35100feb10 2023-07-14T19:03:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.485+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351019d930 con 0x7f35100feb10 2023-07-14T19:03:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.485+0000 7f350e7fc700 1 -- 172.21.15.78:0/3371001201 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f350000f040 con 0x7f35100feb10 2023-07-14T19:03:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.485+0000 7f350e7fc700 1 -- 172.21.15.78:0/3371001201 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500004280 con 0x7f35100feb10 2023-07-14T19:03:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.485+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351019de40 con 0x7f35100feb10 2023-07-14T19:03:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.486+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3510106090 con 0x7f35100feb10 2023-07-14T19:03:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.487+0000 7f350e7fc700 1 -- 172.21.15.78:0/3371001201 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f350000cad0 con 0x7f35100feb10 2023-07-14T19:03:52.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.487+0000 7f350e7fc700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34f80a11e0 0x7f34f80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.488+0000 7f350e7fc700 1 -- 172.21.15.78:0/3371001201 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f35000cce10 con 0x7f35100feb10 2023-07-14T19:03:52.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.488+0000 7f3514da0700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34f80a11e0 0x7f34f80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.488+0000 7f3514da0700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34f80a11e0 0x7f34f80a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f35100680a0 tx=0x7f3504009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:52.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.492+0000 7f350e7fc700 1 -- 172.21.15.78:0/3371001201 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3500096180 con 0x7f35100feb10 2023-07-14T19:03:52.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:52 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1666545714' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:52.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:52 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3052695373' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:03:52.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.681+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f351002cb10 con 0x7f35100feb10 2023-07-14T19:03:52.685 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.682+0000 7f350e7fc700 1 -- 172.21.15.78:0/3371001201 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f35000997a0 con 0x7f35100feb10 2023-07-14T19:03:52.685 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-07-14T19:03:52.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.683+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34f80a11e0 msgr2=0x7f34f80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.683+0000 7f3517805700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34f80a11e0 0x7f34f80a3690 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f35100680a0 tx=0x7f3504009260 comp rx=0 tx=0).stop 2023-07-14T19:03:52.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35100feb10 msgr2=0x7f3510068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35100feb10 0x7f3510068330 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f350000efc0 tx=0x7f350000c3a0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 shutdown_connections 2023-07-14T19:03:52.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f34f80a11e0 0x7f34f80a3690 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3510100690 0x7f351019cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f35100feb10 0x7f3510068330 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 --2- 172.21.15.78:0/3371001201 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35100ffd00 0x7f351019a560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 >> 172.21.15.78:0/3371001201 conn(0x7f35100fa130 msgr2=0x7f35101031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:52.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 shutdown_connections 2023-07-14T19:03:52.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.684+0000 7f3517805700 1 -- 172.21.15.78:0/3371001201 wait complete. 2023-07-14T19:03:52.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T19:03:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 -- 172.21.15.78:0/3536783718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c0fdc90 msgr2=0x7f6c0c0fe0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 --2- 172.21.15.78:0/3536783718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c0fdc90 0x7f6c0c0fe0e0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f6c08009990 tx=0x7f6c08009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 -- 172.21.15.78:0/3536783718 shutdown_connections 2023-07-14T19:03:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 --2- 172.21.15.78:0/3536783718 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c0c100110 0x7f6c0c100520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.825 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 --2- 172.21.15.78:0/3536783718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c0c0fe620 0x7f6c0c0feac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 --2- 172.21.15.78:0/3536783718 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c0fdc90 0x7f6c0c0fe0e0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 -- 172.21.15.78:0/3536783718 >> 172.21.15.78:0/3536783718 conn(0x7f6c0c0f98a0 msgr2=0x7f6c0c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 -- 172.21.15.78:0/3536783718 shutdown_connections 2023-07-14T19:03:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 -- 172.21.15.78:0/3536783718 wait complete. 2023-07-14T19:03:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 Processor -- start 2023-07-14T19:03:52.826 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.821+0000 7f6c14513700 1 -- start start 2023-07-14T19:03:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c14513700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c0c0fdc90 0x7f6c0c197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c14513700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c0c0fe620 0x7f6c0c197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c14513700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c100110 0x7f6c0c19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c14513700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19f5f0 con 0x7f6c0c100110 2023-07-14T19:03:52.828 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c14513700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19f730 con 0x7f6c0c0fdc90 2023-07-14T19:03:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c14513700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0c19f8c0 con 0x7f6c0c0fe620 2023-07-14T19:03:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c100110 0x7f6c0c19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c100110 0x7f6c0c19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35592/0 (socket says 172.21.15.78:35592) 2023-07-14T19:03:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 -- 172.21.15.78:0/2662677022 learned_addr learned my addr 172.21.15.78:0/2662677022 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c11aae700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c0c0fe620 0x7f6c0c197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c122af700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c0c0fdc90 0x7f6c0c197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 -- 172.21.15.78:0/2662677022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c0c0fe620 msgr2=0x7f6c0c197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.829 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c0c0fe620 0x7f6c0c197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 -- 172.21.15.78:0/2662677022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c0c0fdc90 msgr2=0x7f6c0c197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c0c0fdc90 0x7f6c0c197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 -- 172.21.15.78:0/2662677022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c080095f0 con 0x7f6c0c100110 2023-07-14T19:03:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c12ab0700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c100110 0x7f6c0c19f020 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6c04007ee0 tx=0x7f6c0400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c037fe700 1 -- 172.21.15.78:0/2662677022 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c0400ddd0 con 0x7f6c0c100110 2023-07-14T19:03:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c037fe700 1 -- 172.21.15.78:0/2662677022 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6c0400f040 con 0x7f6c0c100110 2023-07-14T19:03:52.830 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c0c19fba0 con 0x7f6c0c100110 2023-07-14T19:03:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c037fe700 1 -- 172.21.15.78:0/2662677022 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c040047d0 con 0x7f6c0c100110 2023-07-14T19:03:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.822+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0c1a0060 con 0x7f6c0c100110 2023-07-14T19:03:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.823+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c0c109b40 con 0x7f6c0c100110 2023-07-14T19:03:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.824+0000 7f6c037fe700 1 -- 172.21.15.78:0/2662677022 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6c04003a00 con 0x7f6c0c100110 2023-07-14T19:03:52.831 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.825+0000 7f6c037fe700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6bf80a11e0 0x7f6bf80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.825+0000 7f6c122af700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6bf80a11e0 0x7f6bf80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.825+0000 7f6c037fe700 1 -- 172.21.15.78:0/2662677022 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6c040cc730 con 0x7f6c0c100110 2023-07-14T19:03:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.825+0000 7f6c122af700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6bf80a11e0 0x7f6bf80a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c067770 tx=0x7f6bfc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:52.832 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:52.829+0000 7f6c037fe700 1 -- 172.21.15.78:0/2662677022 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6c0404e780 con 0x7f6c0c100110 2023-07-14T19:03:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.017+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f6c0c02cb30 con 0x7f6c0c100110 2023-07-14T19:03:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.018+0000 7f6c037fe700 1 -- 172.21.15.78:0/2662677022 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f6c04095a10 con 0x7f6c0c100110 2023-07-14T19:03:53.021 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:03:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6bf80a11e0 msgr2=0x7f6bf80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6bf80a11e0 0x7f6bf80a3690 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6c0c067770 tx=0x7f6bfc009260 comp rx=0 tx=0).stop 2023-07-14T19:03:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c100110 msgr2=0x7f6c0c19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c100110 0x7f6c0c19f020 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f6c04007ee0 tx=0x7f6c0400d3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 shutdown_connections 2023-07-14T19:03:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6bf80a11e0 0x7f6bf80a3690 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c0c0fdc90 0x7f6c0c197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c0c100110 0x7f6c0c19f020 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 --2- 172.21.15.78:0/2662677022 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c0c0fe620 0x7f6c0c197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 >> 172.21.15.78:0/2662677022 conn(0x7f6c0c0f98a0 msgr2=0x7f6c0c0fb620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 shutdown_connections 2023-07-14T19:03:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.020+0000 7f6c14513700 1 -- 172.21.15.78:0/2662677022 wait complete. 2023-07-14T19:03:53.035 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:03:53.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 -- 172.21.15.78:0/1887696519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54001006e0 msgr2=0x7f5400102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.159 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 --2- 172.21.15.78:0/1887696519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54001006e0 0x7f5400102ac0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f53f0009990 tx=0x7f53f0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 -- 172.21.15.78:0/1887696519 shutdown_connections 2023-07-14T19:03:53.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 --2- 172.21.15.78:0/1887696519 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5400105920 0x7f5400107d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.160 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 --2- 172.21.15.78:0/1887696519 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f54001006e0 0x7f5400102ac0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 --2- 172.21.15.78:0/1887696519 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5400103000 0x7f54001053e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 -- 172.21.15.78:0/1887696519 >> 172.21.15.78:0/1887696519 conn(0x7f54000fa100 msgr2=0x7f54000fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:53.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 -- 172.21.15.78:0/1887696519 shutdown_connections 2023-07-14T19:03:53.161 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.152+0000 7f5405dfd700 1 -- 172.21.15.78:0/1887696519 wait complete. 2023-07-14T19:03:53.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f5405dfd700 1 Processor -- start 2023-07-14T19:03:53.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f5405dfd700 1 -- start start 2023-07-14T19:03:53.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f5405dfd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54001006e0 0x7f5400195870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.162 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f5405dfd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5400103000 0x7f5400195db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f5405dfd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5400105920 0x7f540019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f5405dfd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540019d3f0 con 0x7f5400103000 2023-07-14T19:03:53.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f5405dfd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540019d530 con 0x7f54001006e0 2023-07-14T19:03:53.163 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f5405dfd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f540019d6c0 con 0x7f5400105920 2023-07-14T19:03:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f53feffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5400103000 0x7f5400195db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f53feffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5400103000 0x7f5400195db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35604/0 (socket says 172.21.15.78:35604) 2023-07-14T19:03:53.164 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f53feffd700 1 -- 172.21.15.78:0/3580737980 learned_addr learned my addr 172.21.15.78:0/3580737980 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:53.165 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f53ff7fe700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54001006e0 0x7f5400195870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.153+0000 7f53fffff700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5400105920 0x7f540019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53feffd700 1 -- 172.21.15.78:0/3580737980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5400105920 msgr2=0x7f540019ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53feffd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5400105920 0x7f540019ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.166 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53feffd700 1 -- 172.21.15.78:0/3580737980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54001006e0 msgr2=0x7f5400195870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53feffd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54001006e0 0x7f5400195870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53feffd700 1 -- 172.21.15.78:0/3580737980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53f00095f0 con 0x7f5400103000 2023-07-14T19:03:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53feffd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5400103000 0x7f5400195db0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f53f400d950 tx=0x7f53f400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:53.167 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53fcff9700 1 -- 172.21.15.78:0/3580737980 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f400f890 con 0x7f5400103000 2023-07-14T19:03:53.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53fcff9700 1 -- 172.21.15.78:0/3580737980 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f53f400fe80 con 0x7f5400103000 2023-07-14T19:03:53.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f540019d9a0 con 0x7f5400103000 2023-07-14T19:03:53.168 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f53fcff9700 1 -- 172.21.15.78:0/3580737980 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53f400e3b0 con 0x7f5400103000 2023-07-14T19:03:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.154+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f540019deb0 con 0x7f5400103000 2023-07-14T19:03:53.169 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.155+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54000fbc70 con 0x7f5400103000 2023-07-14T19:03:53.170 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.156+0000 7f53fcff9700 1 -- 172.21.15.78:0/3580737980 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f53f4010460 con 0x7f5400103000 2023-07-14T19:03:53.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.156+0000 7f53fcff9700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53e80a1220 0x7f53e80a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.157+0000 7f53fcff9700 1 -- 172.21.15.78:0/3580737980 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f53f40ccb20 con 0x7f5400103000 2023-07-14T19:03:53.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.157+0000 7f53ff7fe700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53e80a1220 0x7f53e80a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.171 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.157+0000 7f53ff7fe700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53e80a1220 0x7f53e80a36d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53f000b5c0 tx=0x7f53f0009f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:53.172 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.160+0000 7f53fcff9700 1 -- 172.21.15.78:0/3580737980 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f53f4095e00 con 0x7f5400103000 2023-07-14T19:03:53.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:53 smithi078 ceph-mon[127425]: pgmap v298: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:53.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:53 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1361137081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:53 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3371001201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:53 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2662677022' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.393+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f540002caf0 con 0x7f5400103000 2023-07-14T19:03:53.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.393+0000 7f53fcff9700 1 -- 172.21.15.78:0/3580737980 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10808 (secure 0 0 0) 0x7f53f4016090 con 0x7f5400103000 2023-07-14T19:03:53.398 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:53.398 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:02:50.605732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24485":{"gid":24485,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:03:53.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53e80a1220 msgr2=0x7f53e80a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53e80a1220 0x7f53e80a36d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f53f000b5c0 tx=0x7f53f0009f00 comp rx=0 tx=0).stop 2023-07-14T19:03:53.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5400103000 msgr2=0x7f5400195db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5400103000 0x7f5400195db0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f53f400d950 tx=0x7f53f400dc60 comp rx=0 tx=0).stop 2023-07-14T19:03:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 shutdown_connections 2023-07-14T19:03:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f53e80a1220 0x7f53e80a36d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f54001006e0 0x7f5400195870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5400105920 0x7f540019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 --2- 172.21.15.78:0/3580737980 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5400103000 0x7f5400195db0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 >> 172.21.15.78:0/3580737980 conn(0x7f54000fa100 msgr2=0x7f5400101340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 shutdown_connections 2023-07-14T19:03:53.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.396+0000 7f5405dfd700 1 -- 172.21.15.78:0/3580737980 wait complete. 2023-07-14T19:03:53.403 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2023-07-14T19:03:53.412 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:03:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:53 smithi088 ceph-mon[128973]: pgmap v298: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:53 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1361137081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:53 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3371001201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:53 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2662677022' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 -- 172.21.15.78:0/3270249730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740feb00 msgr2=0x7fd8740fef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 --2- 172.21.15.78:0/3270249730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740feb00 0x7fd8740fef10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fd86c009990 tx=0x7fd86c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 -- 172.21.15.78:0/3270249730 shutdown_connections 2023-07-14T19:03:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 --2- 172.21.15.78:0/3270249730 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd874100680 0x7fd874106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 --2- 172.21.15.78:0/3270249730 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740feb00 0x7fd8740fef10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 --2- 172.21.15.78:0/3270249730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8740ffcf0 0x7fd874100140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 -- 172.21.15.78:0/3270249730 >> 172.21.15.78:0/3270249730 conn(0x7fd8740fa100 msgr2=0x7fd8740fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:53.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 -- 172.21.15.78:0/3270249730 shutdown_connections 2023-07-14T19:03:53.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 -- 172.21.15.78:0/3270249730 wait complete. 2023-07-14T19:03:53.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.530+0000 7fd879ba8700 1 Processor -- start 2023-07-14T19:03:53.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd879ba8700 1 -- start start 2023-07-14T19:03:53.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd879ba8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8740feb00 0x7fd874194700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd879ba8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740ffcf0 0x7fd874194c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd879ba8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd874100680 0x7fd87419bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd879ba8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87419c280 con 0x7fd8740ffcf0 2023-07-14T19:03:53.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd879ba8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87419c3c0 con 0x7fd874100680 2023-07-14T19:03:53.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd879ba8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd87419c550 con 0x7fd8740feb00 2023-07-14T19:03:53.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740ffcf0 0x7fd874194c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740ffcf0 0x7fd874194c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35618/0 (socket says 172.21.15.78:35618) 2023-07-14T19:03:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 -- 172.21.15.78:0/1252437966 learned_addr learned my addr 172.21.15.78:0/1252437966 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd8737fe700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8740feb00 0x7fd874194700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd873fff700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd874100680 0x7fd87419bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 -- 172.21.15.78:0/1252437966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8740feb00 msgr2=0x7fd874194700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8740feb00 0x7fd874194700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 -- 172.21.15.78:0/1252437966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd874100680 msgr2=0x7fd87419bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd874100680 0x7fd87419bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 -- 172.21.15.78:0/1252437966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86c0095f0 con 0x7fd8740ffcf0 2023-07-14T19:03:53.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd8737fe700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8740feb00 0x7fd874194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:53.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd873fff700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd874100680 0x7fd87419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:53.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.531+0000 7fd872ffd700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740ffcf0 0x7fd874194c40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd85c007b50 tx=0x7fd85c007f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:53.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.532+0000 7fd870ff9700 1 -- 172.21.15.78:0/1252437966 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c011070 con 0x7fd8740ffcf0 2023-07-14T19:03:53.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.532+0000 7fd870ff9700 1 -- 172.21.15.78:0/1252437966 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd85c00c2e0 con 0x7fd8740ffcf0 2023-07-14T19:03:53.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.532+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd874067800 con 0x7fd8740ffcf0 2023-07-14T19:03:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.532+0000 7fd870ff9700 1 -- 172.21.15.78:0/1252437966 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c00e5c0 con 0x7fd8740ffcf0 2023-07-14T19:03:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.532+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd874067d60 con 0x7fd8740ffcf0 2023-07-14T19:03:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.533+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd87410a1d0 con 0x7fd8740ffcf0 2023-07-14T19:03:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.534+0000 7fd870ff9700 1 -- 172.21.15.78:0/1252437966 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd85c004e50 con 0x7fd8740ffcf0 2023-07-14T19:03:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.534+0000 7fd870ff9700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8600a12a0 0x7fd8600a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.534+0000 7fd8737fe700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8600a12a0 0x7fd8600a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.535+0000 7fd870ff9700 1 -- 172.21.15.78:0/1252437966 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd85c0ce6a0 con 0x7fd8740ffcf0 2023-07-14T19:03:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.535+0000 7fd8737fe700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8600a12a0 0x7fd8600a3750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00b5c0 tx=0x7fd86c005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:53.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.538+0000 7fd870ff9700 1 -- 172.21.15.78:0/1252437966 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd85c0979b0 con 0x7fd8740ffcf0 2023-07-14T19:03:53.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:53 smithi102 ceph-mon[129280]: pgmap v298: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:53.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:53 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1361137081' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:53 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3371001201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:53 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2662677022' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:03:53.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.729+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd87410a3c0 con 0x7fd8740ffcf0 2023-07-14T19:03:53.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.731+0000 7fd870ff9700 1 -- 172.21.15.78:0/1252437966 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24453 (secure 0 0 0) 0x7fd85c09afd0 con 0x7fd8740ffcf0 2023-07-14T19:03:53.734 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:53.735 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:03:31.201384+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:03:53.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8600a12a0 msgr2=0x7fd8600a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8600a12a0 0x7fd8600a3750 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00b5c0 tx=0x7fd86c005fb0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740ffcf0 msgr2=0x7fd874194c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740ffcf0 0x7fd874194c40 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fd85c007b50 tx=0x7fd85c007f10 comp rx=0 tx=0).stop 2023-07-14T19:03:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 shutdown_connections 2023-07-14T19:03:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd8600a12a0 0x7fd8600a3750 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd874100680 0x7fd87419bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8740feb00 0x7fd874194700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 --2- 172.21.15.78:0/1252437966 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd8740ffcf0 0x7fd874194c40 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 >> 172.21.15.78:0/1252437966 conn(0x7fd8740fa100 msgr2=0x7fd8740fc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 shutdown_connections 2023-07-14T19:03:53.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.733+0000 7fd879ba8700 1 -- 172.21.15.78:0/1252437966 wait complete. 2023-07-14T19:03:53.749 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:02:28.269050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:02:29.547771+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T19:02:32.797605+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T19:03:53.750 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:03:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 -- 172.21.15.78:0/1478459322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b41059a0 msgr2=0x7f93b4107da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 --2- 172.21.15.78:0/1478459322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b41059a0 0x7f93b4107da0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f93ac00b0a0 tx=0x7f93ac00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 -- 172.21.15.78:0/1478459322 shutdown_connections 2023-07-14T19:03:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 --2- 172.21.15.78:0/1478459322 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b4100760 0x7f93b4102b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.874 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 --2- 172.21.15.78:0/1478459322 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b41059a0 0x7f93b4107da0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 --2- 172.21.15.78:0/1478459322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b4103080 0x7f93b4105460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 -- 172.21.15.78:0/1478459322 >> 172.21.15.78:0/1478459322 conn(0x7f93b40fa1a0 msgr2=0x7f93b40fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 -- 172.21.15.78:0/1478459322 shutdown_connections 2023-07-14T19:03:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.870+0000 7f93b9502700 1 -- 172.21.15.78:0/1478459322 wait complete. 2023-07-14T19:03:53.875 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b9502700 1 Processor -- start 2023-07-14T19:03:53.877 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b9502700 1 -- start start 2023-07-14T19:03:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b9502700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b4100760 0x7f93b4195940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b9502700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b4103080 0x7f93b4195e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b9502700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b41059a0 0x7f93b419cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b9502700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b419d4c0 con 0x7f93b4103080 2023-07-14T19:03:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b9502700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b419d600 con 0x7f93b4100760 2023-07-14T19:03:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b9502700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b419d790 con 0x7f93b41059a0 2023-07-14T19:03:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b4103080 0x7f93b4195e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.878 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b27fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b4103080 0x7f93b4195e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35632/0 (socket says 172.21.15.78:35632) 2023-07-14T19:03:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b27fc700 1 -- 172.21.15.78:0/3953194704 learned_addr learned my addr 172.21.15.78:0/3953194704 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b2ffd700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b4100760 0x7f93b4195940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b27fc700 1 -- 172.21.15.78:0/3953194704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b41059a0 msgr2=0x7f93b419cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b37fe700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b41059a0 0x7f93b419cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b27fc700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b41059a0 0x7f93b419cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b27fc700 1 -- 172.21.15.78:0/3953194704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b4100760 msgr2=0x7f93b4195940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:53.879 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b27fc700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b4100760 0x7f93b4195940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.871+0000 7f93b27fc700 1 -- 172.21.15.78:0/3953194704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac009d00 con 0x7f93b4103080 2023-07-14T19:03:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.872+0000 7f93b27fc700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b4103080 0x7f93b4195e80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c920 tx=0x7f939c00cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.872+0000 7f93abfff700 1 -- 172.21.15.78:0/3953194704 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c007b00 con 0x7f93b4103080 2023-07-14T19:03:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.872+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b419da70 con 0x7f93b4103080 2023-07-14T19:03:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.872+0000 7f93abfff700 1 -- 172.21.15.78:0/3953194704 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f939c00ce70 con 0x7f93b4103080 2023-07-14T19:03:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.872+0000 7f93abfff700 1 -- 172.21.15.78:0/3953194704 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f939c0053e0 con 0x7f93b4103080 2023-07-14T19:03:53.880 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.872+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b419dfd0 con 0x7f93b4103080 2023-07-14T19:03:53.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.873+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93b40fbd10 con 0x7f93b4103080 2023-07-14T19:03:53.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.874+0000 7f93abfff700 1 -- 172.21.15.78:0/3953194704 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f939c004750 con 0x7f93b4103080 2023-07-14T19:03:53.881 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.874+0000 7f93abfff700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93a00a11e0 0x7f93a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:53.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.874+0000 7f93b2ffd700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93a00a11e0 0x7f93a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:53.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.874+0000 7f93abfff700 1 -- 172.21.15.78:0/3953194704 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f939c0ccdb0 con 0x7f93b4103080 2023-07-14T19:03:53.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.875+0000 7f93b2ffd700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93a00a11e0 0x7f93a00a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f93a4007c60 tx=0x7f93a40073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:53.882 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:53.878+0000 7f93abfff700 1 -- 172.21.15.78:0/3953194704 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f939c096090 con 0x7f93b4103080 2023-07-14T19:03:54.069 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.066+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f93b402caf0 con 0x7f93b4103080 2023-07-14T19:03:54.070 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.066+0000 7f93abfff700 1 -- 172.21.15.78:0/3953194704 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f939c01d020 con 0x7f93b4103080 2023-07-14T19:03:54.071 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:03:54.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93a00a11e0 msgr2=0x7f93a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93a00a11e0 0x7f93a00a3690 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f93a4007c60 tx=0x7f93a40073d0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b4103080 msgr2=0x7f93b4195e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b4103080 0x7f93b4195e80 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f939c00c920 tx=0x7f939c00cc30 comp rx=0 tx=0).stop 2023-07-14T19:03:54.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 shutdown_connections 2023-07-14T19:03:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93a00a11e0 0x7f93a00a3690 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93b4100760 0x7f93b4195940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b41059a0 0x7f93b419cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 --2- 172.21.15.78:0/3953194704 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93b4103080 0x7f93b4195e80 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 >> 172.21.15.78:0/3953194704 conn(0x7f93b40fa1a0 msgr2=0x7f93b41013c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.068+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 shutdown_connections 2023-07-14T19:03:54.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.069+0000 7f93b9502700 1 -- 172.21.15.78:0/3953194704 wait complete. 2023-07-14T19:03:54.084 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:03:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 -- 172.21.15.78:0/938854641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 msgr2=0x7f5ed0106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 --2- 172.21.15.78:0/938854641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 0x7f5ed0106f50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f5ec800b0a0 tx=0x7f5ec800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 -- 172.21.15.78:0/938854641 shutdown_connections 2023-07-14T19:03:54.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 --2- 172.21.15.78:0/938854641 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ed00feb10 0x7f5ed00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 --2- 172.21.15.78:0/938854641 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 0x7f5ed0106f50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 --2- 172.21.15.78:0/938854641 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ed00ffd00 0x7f5ed0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 -- 172.21.15.78:0/938854641 >> 172.21.15.78:0/938854641 conn(0x7f5ed00fa130 msgr2=0x7f5ed00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 -- 172.21.15.78:0/938854641 shutdown_connections 2023-07-14T19:03:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.207+0000 7f5ed6211700 1 -- 172.21.15.78:0/938854641 wait complete. 2023-07-14T19:03:54.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ed6211700 1 Processor -- start 2023-07-14T19:03:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ed6211700 1 -- start start 2023-07-14T19:03:54.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ed6211700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ed00feb10 0x7f5ed0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ed6211700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ed00ffd00 0x7f5ed019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ed6211700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 0x7f5ed019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ed6211700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed019d4a0 con 0x7f5ed0100690 2023-07-14T19:03:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ed6211700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed019d610 con 0x7f5ed00ffd00 2023-07-14T19:03:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ed6211700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ed019d7d0 con 0x7f5ed00feb10 2023-07-14T19:03:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 0x7f5ed019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecffff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 0x7f5ed019cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35648/0 (socket says 172.21.15.78:35648) 2023-07-14T19:03:54.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecffff700 1 -- 172.21.15.78:0/2138671683 learned_addr learned my addr 172.21.15.78:0/2138671683 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5eceffd700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ed00ffd00 0x7f5ed019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecf7fe700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ed00feb10 0x7f5ed0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecffff700 1 -- 172.21.15.78:0/2138671683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ed00feb10 msgr2=0x7f5ed0069340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecffff700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ed00feb10 0x7f5ed0069340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecffff700 1 -- 172.21.15.78:0/2138671683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ed00ffd00 msgr2=0x7f5ed019a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecffff700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ed00ffd00 0x7f5ed019a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecffff700 1 -- 172.21.15.78:0/2138671683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec8009d00 con 0x7f5ed0100690 2023-07-14T19:03:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.208+0000 7f5ecf7fe700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ed00feb10 0x7f5ed0069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.209+0000 7f5ecffff700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 0x7f5ed019cd00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80095e0 tx=0x7f5ec800ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.209+0000 7f5eccff9700 1 -- 172.21.15.78:0/2138671683 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec800e040 con 0x7f5ed0100690 2023-07-14T19:03:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.209+0000 7f5eccff9700 1 -- 172.21.15.78:0/2138671683 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f5ec8003c00 con 0x7f5ed0100690 2023-07-14T19:03:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.209+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ed019da50 con 0x7f5ed0100690 2023-07-14T19:03:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.209+0000 7f5eccff9700 1 -- 172.21.15.78:0/2138671683 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ec80052a0 con 0x7f5ed0100690 2023-07-14T19:03:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.209+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ed019df20 con 0x7f5ed0100690 2023-07-14T19:03:54.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.210+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ed0105a90 con 0x7f5ed0100690 2023-07-14T19:03:54.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.211+0000 7f5eccff9700 1 -- 172.21.15.78:0/2138671683 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f5ec801d470 con 0x7f5ed0100690 2023-07-14T19:03:54.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.211+0000 7f5eccff9700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb80a1260 0x7f5eb80a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.211+0000 7f5eccff9700 1 -- 172.21.15.78:0/2138671683 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f5ec80cdfd0 con 0x7f5ed0100690 2023-07-14T19:03:54.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.211+0000 7f5ecf7fe700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb80a1260 0x7f5eb80a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.212+0000 7f5ecf7fe700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb80a1260 0x7f5eb80a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00ffb60 tx=0x7f5ec0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:54.219 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.215+0000 7f5eccff9700 1 -- 172.21.15.78:0/2138671683 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f5ec8100780 con 0x7f5ed0100690 2023-07-14T19:03:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.403+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ed002caf0 con 0x7f5ed0100690 2023-07-14T19:03:54.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.403+0000 7f5eccff9700 1 -- 172.21.15.78:0/2138671683 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5ec8097340 con 0x7f5ed0100690 2023-07-14T19:03:54.407 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:03:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb80a1260 msgr2=0x7f5eb80a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb80a1260 0x7f5eb80a3710 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f5ed00ffb60 tx=0x7f5ec0009260 comp rx=0 tx=0).stop 2023-07-14T19:03:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 msgr2=0x7f5ed019cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 0x7f5ed019cd00 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f5ec80095e0 tx=0x7f5ec800ba50 comp rx=0 tx=0).stop 2023-07-14T19:03:54.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 shutdown_connections 2023-07-14T19:03:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f5eb80a1260 0x7f5eb80a3710 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f5ed0100690 0x7f5ed019cd00 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ed00feb10 0x7f5ed0069340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 --2- 172.21.15.78:0/2138671683 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f5ed00ffd00 0x7f5ed019a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 >> 172.21.15.78:0/2138671683 conn(0x7f5ed00fa130 msgr2=0x7f5ed00fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 shutdown_connections 2023-07-14T19:03:54.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.405+0000 7f5ed6211700 1 -- 172.21.15.78:0/2138671683 wait complete. 2023-07-14T19:03:54.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:54 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3580737980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:54.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:54 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1252437966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:54.420 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:54 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3953194704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:03:54.422 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:03:54.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:54 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3580737980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:54.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:54 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1252437966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:54.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:54 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3953194704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:03:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.543+0000 7fed1f758700 1 -- 172.21.15.78:0/2671489073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed180feb30 msgr2=0x7fed180fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.543+0000 7fed1f758700 1 --2- 172.21.15.78:0/2671489073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed180feb30 0x7fed180fef40 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fed0c009990 tx=0x7fed0c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.543+0000 7fed1f758700 1 -- 172.21.15.78:0/2671489073 shutdown_connections 2023-07-14T19:03:54.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.543+0000 7fed1f758700 1 --2- 172.21.15.78:0/2671489073 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed18100770 0x7fed18107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.543+0000 7fed1f758700 1 --2- 172.21.15.78:0/2671489073 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed180feb30 0x7fed180fef40 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.543+0000 7fed1f758700 1 --2- 172.21.15.78:0/2671489073 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed180ffdc0 0x7fed18100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.543+0000 7fed1f758700 1 -- 172.21.15.78:0/2671489073 >> 172.21.15.78:0/2671489073 conn(0x7fed180fa150 msgr2=0x7fed180fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 -- 172.21.15.78:0/2671489073 shutdown_connections 2023-07-14T19:03:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 -- 172.21.15.78:0/2671489073 wait complete. 2023-07-14T19:03:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 Processor -- start 2023-07-14T19:03:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 -- start start 2023-07-14T19:03:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed180feb30 0x7fed18194850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed180ffdc0 0x7fed18194d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100770 0x7fed1819be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1819c3d0 con 0x7fed18100770 2023-07-14T19:03:54.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1819c510 con 0x7fed180ffdc0 2023-07-14T19:03:54.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1f758700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1819c6a0 con 0x7fed180feb30 2023-07-14T19:03:54.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1dcf5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100770 0x7fed1819be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1dcf5700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100770 0x7fed1819be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35660/0 (socket says 172.21.15.78:35660) 2023-07-14T19:03:54.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1dcf5700 1 -- 172.21.15.78:0/3698513066 learned_addr learned my addr 172.21.15.78:0/3698513066 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:54.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.544+0000 7fed1ccf3700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed180ffdc0 0x7fed18194d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.556 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1d4f4700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed180feb30 0x7fed18194850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1dcf5700 1 -- 172.21.15.78:0/3698513066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed180feb30 msgr2=0x7fed18194850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1dcf5700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed180feb30 0x7fed18194850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.557 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1dcf5700 1 -- 172.21.15.78:0/3698513066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed180ffdc0 msgr2=0x7fed18194d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1dcf5700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed180ffdc0 0x7fed18194d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1dcf5700 1 -- 172.21.15.78:0/3698513066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c0095f0 con 0x7fed18100770 2023-07-14T19:03:54.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1ccf3700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed180ffdc0 0x7fed18194d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:54.558 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1dcf5700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100770 0x7fed1819be00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fed1000ada0 tx=0x7fed1000c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:54.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed0a7fc700 1 -- 172.21.15.78:0/3698513066 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed1000f070 con 0x7fed18100770 2023-07-14T19:03:54.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed0a7fc700 1 -- 172.21.15.78:0/3698513066 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed10007b30 con 0x7fed18100770 2023-07-14T19:03:54.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed0a7fc700 1 -- 172.21.15.78:0/3698513066 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed100053e0 con 0x7fed18100770 2023-07-14T19:03:54.559 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1819c980 con 0x7fed18100770 2023-07-14T19:03:54.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.545+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1819ce40 con 0x7fed18100770 2023-07-14T19:03:54.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.546+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed181062c0 con 0x7fed18100770 2023-07-14T19:03:54.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.547+0000 7fed0a7fc700 1 -- 172.21.15.78:0/3698513066 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fed1000c980 con 0x7fed18100770 2023-07-14T19:03:54.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.548+0000 7fed0a7fc700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed040a1260 0x7fed040a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.548+0000 7fed1d4f4700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed040a1260 0x7fed040a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.548+0000 7fed0a7fc700 1 -- 172.21.15.78:0/3698513066 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fed100cca60 con 0x7fed18100770 2023-07-14T19:03:54.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.548+0000 7fed1d4f4700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed040a1260 0x7fed040a3710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fed0c006010 tx=0x7fed0c00bf60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:54.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.551+0000 7fed0a7fc700 1 -- 172.21.15.78:0/3698513066 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fed10095d70 con 0x7fed18100770 2023-07-14T19:03:54.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:54 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3580737980' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:54.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:54 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1252437966' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:54.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:54 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3953194704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:03:54.746 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.743+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fed1802ca70 con 0x7fed18100770 2023-07-14T19:03:54.747 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.743+0000 7fed0a7fc700 1 -- 172.21.15.78:0/3698513066 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7fed1000cc30 con 0x7fed18100770 2023-07-14T19:03:54.747 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-07-14T19:03:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed040a1260 msgr2=0x7fed040a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed040a1260 0x7fed040a3710 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fed0c006010 tx=0x7fed0c00bf60 comp rx=0 tx=0).stop 2023-07-14T19:03:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100770 msgr2=0x7fed1819be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.749 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100770 0x7fed1819be00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fed1000ada0 tx=0x7fed1000c330 comp rx=0 tx=0).stop 2023-07-14T19:03:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 shutdown_connections 2023-07-14T19:03:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed040a1260 0x7fed040a3710 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed18100770 0x7fed1819be00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed180feb30 0x7fed18194850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 --2- 172.21.15.78:0/3698513066 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed180ffdc0 0x7fed18194d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 >> 172.21.15.78:0/3698513066 conn(0x7fed180fa150 msgr2=0x7fed18103400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 shutdown_connections 2023-07-14T19:03:54.750 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.745+0000 7fed1f758700 1 -- 172.21.15.78:0/3698513066 wait complete. 2023-07-14T19:03:54.761 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T19:03:54.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 -- 172.21.15.78:0/825160685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa440100690 msgr2=0x7fa440106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 --2- 172.21.15.78:0/825160685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa440100690 0x7fa440106f50 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa43800b0a0 tx=0x7fa43800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.886 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 -- 172.21.15.78:0/825160685 shutdown_connections 2023-07-14T19:03:54.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 --2- 172.21.15.78:0/825160685 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa440100690 0x7fa440106f50 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 --2- 172.21.15.78:0/825160685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4400feb10 0x7fa4400fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 --2- 172.21.15.78:0/825160685 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa4400ffd00 0x7fa440100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 -- 172.21.15.78:0/825160685 >> 172.21.15.78:0/825160685 conn(0x7fa4400fa130 msgr2=0x7fa4400fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:54.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 -- 172.21.15.78:0/825160685 shutdown_connections 2023-07-14T19:03:54.887 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 -- 172.21.15.78:0/825160685 wait complete. 2023-07-14T19:03:54.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 Processor -- start 2023-07-14T19:03:54.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 -- start start 2023-07-14T19:03:54.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.881+0000 7fa445881700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4400feb10 0x7fa440068370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa445881700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4400ffd00 0x7fa44019a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa445881700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa440100690 0x7fa44019cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.888 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa445881700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44019d510 con 0x7fa4400feb10 2023-07-14T19:03:54.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa445881700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44019d680 con 0x7fa440100690 2023-07-14T19:03:54.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa445881700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44019d840 con 0x7fa4400ffd00 2023-07-14T19:03:54.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4400feb10 0x7fa440068370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4400feb10 0x7fa440068370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35670/0 (socket says 172.21.15.78:35670) 2023-07-14T19:03:54.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 -- 172.21.15.78:0/3738529973 learned_addr learned my addr 172.21.15.78:0/3738529973 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:54.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43e7fc700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4400ffd00 0x7fa44019a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43f7fe700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa440100690 0x7fa44019cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.889 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 -- 172.21.15.78:0/3738529973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4400ffd00 msgr2=0x7fa44019a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4400ffd00 0x7fa44019a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 -- 172.21.15.78:0/3738529973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa440100690 msgr2=0x7fa44019cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:54.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa440100690 0x7fa44019cd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:54.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 -- 172.21.15.78:0/3738529973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa438009d00 con 0x7fa4400feb10 2023-07-14T19:03:54.890 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43e7fc700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4400ffd00 0x7fa44019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa43effd700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4400feb10 0x7fa440068370 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa43400d980 tx=0x7fa43400dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa44487f700 1 -- 172.21.15.78:0/3738529973 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43400de60 con 0x7fa4400feb10 2023-07-14T19:03:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa44487f700 1 -- 172.21.15.78:0/3738529973 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa4340044b0 con 0x7fa4400feb10 2023-07-14T19:03:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa44487f700 1 -- 172.21.15.78:0/3738529973 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa434003b40 con 0x7fa4400feb10 2023-07-14T19:03:54.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa44019db20 con 0x7fa4400feb10 2023-07-14T19:03:54.892 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.882+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa44019e030 con 0x7fa4400feb10 2023-07-14T19:03:54.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.883+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa440105a90 con 0x7fa4400feb10 2023-07-14T19:03:54.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.884+0000 7fa44487f700 1 -- 172.21.15.78:0/3738529973 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fa434004870 con 0x7fa4400feb10 2023-07-14T19:03:54.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.885+0000 7fa44487f700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa4280a1260 0x7fa4280a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:54.893 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.885+0000 7fa43e7fc700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa4280a1260 0x7fa4280a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:54.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.885+0000 7fa44487f700 1 -- 172.21.15.78:0/3738529973 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa4340ccde0 con 0x7fa4400feb10 2023-07-14T19:03:54.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.885+0000 7fa43e7fc700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa4280a1260 0x7fa4280a3710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa430009cb0 tx=0x7fa430009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:54.894 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:54.888+0000 7fa44487f700 1 -- 172.21.15.78:0/3738529973 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa4340960e0 con 0x7fa4400feb10 2023-07-14T19:03:55.079 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.076+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fa44002cb50 con 0x7fa4400feb10 2023-07-14T19:03:55.080 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.076+0000 7fa44487f700 1 -- 172.21.15.78:0/3738529973 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fa4340170a0 con 0x7fa4400feb10 2023-07-14T19:03:55.080 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:03:55.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa4280a1260 msgr2=0x7fa4280a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa4280a1260 0x7fa4280a3710 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa430009cb0 tx=0x7fa430009260 comp rx=0 tx=0).stop 2023-07-14T19:03:55.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4400feb10 msgr2=0x7fa440068370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.082 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4400feb10 0x7fa440068370 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa43400d980 tx=0x7fa43400dd40 comp rx=0 tx=0).stop 2023-07-14T19:03:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 shutdown_connections 2023-07-14T19:03:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa4280a1260 0x7fa4280a3710 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa440100690 0x7fa44019cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa4400feb10 0x7fa440068370 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 --2- 172.21.15.78:0/3738529973 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa4400ffd00 0x7fa44019a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.078+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 >> 172.21.15.78:0/3738529973 conn(0x7fa4400fa130 msgr2=0x7fa4400fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.079+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 shutdown_connections 2023-07-14T19:03:55.083 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.079+0000 7fa445881700 1 -- 172.21.15.78:0/3738529973 wait complete. 2023-07-14T19:03:55.094 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:03:55.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 -- 172.21.15.78:0/913513454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 msgr2=0x7f39b0100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 --2- 172.21.15.78:0/913513454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 0x7f39b0100140 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f39ac009990 tx=0x7f39ac009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 -- 172.21.15.78:0/913513454 shutdown_connections 2023-07-14T19:03:55.220 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 --2- 172.21.15.78:0/913513454 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39b00feb00 0x7f39b00fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 --2- 172.21.15.78:0/913513454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b0100680 0x7f39b0106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 --2- 172.21.15.78:0/913513454 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 0x7f39b0100140 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 -- 172.21.15.78:0/913513454 >> 172.21.15.78:0/913513454 conn(0x7f39b00fa100 msgr2=0x7f39b00fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 -- 172.21.15.78:0/913513454 shutdown_connections 2023-07-14T19:03:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.212+0000 7f39b770c700 1 -- 172.21.15.78:0/913513454 wait complete. 2023-07-14T19:03:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b770c700 1 Processor -- start 2023-07-14T19:03:55.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b770c700 1 -- start start 2023-07-14T19:03:55.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b770c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b00feb00 0x7f39b0195810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b770c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 0x7f39b0195d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b770c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39b0100680 0x7f39b019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b770c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b019d390 con 0x7f39b00ffcf0 2023-07-14T19:03:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b770c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b019d4d0 con 0x7f39b0100680 2023-07-14T19:03:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b770c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b019d660 con 0x7f39b00feb00 2023-07-14T19:03:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 0x7f39b0195d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 0x7f39b0195d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35686/0 (socket says 172.21.15.78:35686) 2023-07-14T19:03:55.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 -- 172.21.15.78:0/918068847 learned_addr learned my addr 172.21.15.78:0/918068847 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b54a8700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b00feb00 0x7f39b0195810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b5ca9700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39b0100680 0x7f39b019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 -- 172.21.15.78:0/918068847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b00feb00 msgr2=0x7f39b0195810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b00feb00 0x7f39b0195810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 -- 172.21.15.78:0/918068847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39b0100680 msgr2=0x7f39b019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39b0100680 0x7f39b019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 -- 172.21.15.78:0/918068847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39ac0095f0 con 0x7f39b00ffcf0 2023-07-14T19:03:55.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39b4ca7700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 0x7f39b0195d50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f39ac003f60 tx=0x7f39ac005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:55.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.213+0000 7f39a67fc700 1 -- 172.21.15.78:0/918068847 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac01d070 con 0x7f39b00ffcf0 2023-07-14T19:03:55.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.214+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b019d8e0 con 0x7f39b00ffcf0 2023-07-14T19:03:55.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.214+0000 7f39a67fc700 1 -- 172.21.15.78:0/918068847 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f39ac00bdd0 con 0x7f39b00ffcf0 2023-07-14T19:03:55.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.214+0000 7f39a67fc700 1 -- 172.21.15.78:0/918068847 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39ac022dd0 con 0x7f39b00ffcf0 2023-07-14T19:03:55.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.214+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b019dd60 con 0x7f39b00ffcf0 2023-07-14T19:03:55.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.215+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b0106c30 con 0x7f39b00ffcf0 2023-07-14T19:03:55.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.216+0000 7f39a67fc700 1 -- 172.21.15.78:0/918068847 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f39ac022570 con 0x7f39b00ffcf0 2023-07-14T19:03:55.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.216+0000 7f39a67fc700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f399c0a1220 0x7f399c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.216+0000 7f39b54a8700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f399c0a1220 0x7f399c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.216+0000 7f39a67fc700 1 -- 172.21.15.78:0/918068847 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f39ac0ce3e0 con 0x7f39b00ffcf0 2023-07-14T19:03:55.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.217+0000 7f39b54a8700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f399c0a1220 0x7f399c0a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f39b00ffb50 tx=0x7f39a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:55.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.220+0000 7f39a67fc700 1 -- 172.21.15.78:0/918068847 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f39ac097750 con 0x7f39b00ffcf0 2023-07-14T19:03:55.456 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.453+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39b002cae0 con 0x7f39b00ffcf0 2023-07-14T19:03:55.457 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.453+0000 7f39a67fc700 1 -- 172.21.15.78:0/918068847 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10808 (secure 0 0 0) 0x7f39ac09ad70 con 0x7f39b00ffcf0 2023-07-14T19:03:55.457 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:55.457 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:02:50.605732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24485":{"gid":24485,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:03:55.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f399c0a1220 msgr2=0x7f399c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f399c0a1220 0x7f399c0a36d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f39b00ffb50 tx=0x7f39a0009260 comp rx=0 tx=0).stop 2023-07-14T19:03:55.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 msgr2=0x7f39b0195d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 0x7f39b0195d50 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f39ac003f60 tx=0x7f39ac005740 comp rx=0 tx=0).stop 2023-07-14T19:03:55.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 shutdown_connections 2023-07-14T19:03:55.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f399c0a1220 0x7f399c0a36d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f39b0100680 0x7f39b019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39b00feb00 0x7f39b0195810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 --2- 172.21.15.78:0/918068847 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f39b00ffcf0 0x7f39b0195d50 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.460 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 >> 172.21.15.78:0/918068847 conn(0x7f39b00fa100 msgr2=0x7f39b00fc380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:55.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 shutdown_connections 2023-07-14T19:03:55.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.456+0000 7f39b770c700 1 -- 172.21.15.78:0/918068847 wait complete. 2023-07-14T19:03:55.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:55 smithi088 ceph-mon[128973]: pgmap v299: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:03:55.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:55 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2138671683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:55 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3698513066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:55 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3738529973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.462 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2023-07-14T19:03:55.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:55 smithi078 ceph-mon[127425]: pgmap v299: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:03:55.470 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:55 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2138671683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:55 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3698513066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.471 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:55 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3738529973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.472 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:03:55.594 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 -- 172.21.15.78:0/1281473170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16240feb40 msgr2=0x7f16240fef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 --2- 172.21.15.78:0/1281473170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16240feb40 0x7f16240fef50 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f1618009990 tx=0x7f1618009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 -- 172.21.15.78:0/1281473170 shutdown_connections 2023-07-14T19:03:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 --2- 172.21.15.78:0/1281473170 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16241006c0 0x7f1624106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 --2- 172.21.15.78:0/1281473170 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16240feb40 0x7f16240fef50 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.595 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 --2- 172.21.15.78:0/1281473170 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16240ffd30 0x7f1624100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 -- 172.21.15.78:0/1281473170 >> 172.21.15.78:0/1281473170 conn(0x7f16240fa140 msgr2=0x7f16240fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 -- 172.21.15.78:0/1281473170 shutdown_connections 2023-07-14T19:03:55.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.591+0000 7f162ae55700 1 -- 172.21.15.78:0/1281473170 wait complete. 2023-07-14T19:03:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f162ae55700 1 Processor -- start 2023-07-14T19:03:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f162ae55700 1 -- start start 2023-07-14T19:03:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f162ae55700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16240feb40 0x7f16241946d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f162ae55700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16240ffd30 0x7f1624194c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f162ae55700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16241006c0 0x7f162419bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f162ae55700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162419c250 con 0x7f16241006c0 2023-07-14T19:03:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f162ae55700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162419c390 con 0x7f16240ffd30 2023-07-14T19:03:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f162ae55700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f162419c520 con 0x7f16240feb40 2023-07-14T19:03:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f16293f2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16241006c0 0x7f162419bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f16293f2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16241006c0 0x7f162419bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35702/0 (socket says 172.21.15.78:35702) 2023-07-14T19:03:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f16293f2700 1 -- 172.21.15.78:0/2079856504 learned_addr learned my addr 172.21.15.78:0/2079856504 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:55.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f1623fff700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16240ffd30 0x7f1624194c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f1628bf1700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16240feb40 0x7f16241946d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f16293f2700 1 -- 172.21.15.78:0/2079856504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16240feb40 msgr2=0x7f16241946d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f16293f2700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16240feb40 0x7f16241946d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f16293f2700 1 -- 172.21.15.78:0/2079856504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16240ffd30 msgr2=0x7f1624194c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f16293f2700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16240ffd30 0x7f1624194c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.601 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.592+0000 7f16293f2700 1 -- 172.21.15.78:0/2079856504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16180095f0 con 0x7f16241006c0 2023-07-14T19:03:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.593+0000 7f16293f2700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16241006c0 0x7f162419bc80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f161c007d70 tx=0x7f161c00a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.593+0000 7f1621ffb700 1 -- 172.21.15.78:0/2079856504 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c00ad40 con 0x7f16241006c0 2023-07-14T19:03:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.593+0000 7f1621ffb700 1 -- 172.21.15.78:0/2079856504 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f161c0128b0 con 0x7f16241006c0 2023-07-14T19:03:55.602 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.593+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f162419c800 con 0x7f16241006c0 2023-07-14T19:03:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.593+0000 7f1621ffb700 1 -- 172.21.15.78:0/2079856504 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f161c0116b0 con 0x7f16241006c0 2023-07-14T19:03:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.593+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f162419ccc0 con 0x7f16241006c0 2023-07-14T19:03:55.603 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.594+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1624105a20 con 0x7f16241006c0 2023-07-14T19:03:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.595+0000 7f1621ffb700 1 -- 172.21.15.78:0/2079856504 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f161c012a20 con 0x7f16241006c0 2023-07-14T19:03:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.595+0000 7f1621ffb700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f16140a1220 0x7f16140a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.596+0000 7f1621ffb700 1 -- 172.21.15.78:0/2079856504 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f161c0cd3b0 con 0x7f16241006c0 2023-07-14T19:03:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.596+0000 7f1628bf1700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f16140a1220 0x7f16140a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.605 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.596+0000 7f1628bf1700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f16140a1220 0x7f16140a36d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1618006010 tx=0x7f1618005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:55.606 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.599+0000 7f1621ffb700 1 -- 172.21.15.78:0/2079856504 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f161c0967a0 con 0x7f16241006c0 2023-07-14T19:03:55.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:55 smithi102 ceph-mon[129280]: pgmap v299: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2023-07-14T19:03:55.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:55 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2138671683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:55 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3698513066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:55 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3738529973' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:03:55.795 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.792+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f162402cb30 con 0x7f16241006c0 2023-07-14T19:03:55.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.793+0000 7f1621ffb700 1 -- 172.21.15.78:0/2079856504 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24453 (secure 0 0 0) 0x7f161c099dc0 con 0x7f16241006c0 2023-07-14T19:03:55.797 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:55.798 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:03:31.201384+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:03:55.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f16140a1220 msgr2=0x7f16140a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f16140a1220 0x7f16140a36d0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1618006010 tx=0x7f1618005940 comp rx=0 tx=0).stop 2023-07-14T19:03:55.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16241006c0 msgr2=0x7f162419bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16241006c0 0x7f162419bc80 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f161c007d70 tx=0x7f161c00a340 comp rx=0 tx=0).stop 2023-07-14T19:03:55.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 shutdown_connections 2023-07-14T19:03:55.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f16140a1220 0x7f16140a36d0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f16241006c0 0x7f162419bc80 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f16240feb40 0x7f16241946d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 --2- 172.21.15.78:0/2079856504 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f16240ffd30 0x7f1624194c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 >> 172.21.15.78:0/2079856504 conn(0x7f16240fa140 msgr2=0x7f16240fc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:55.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 shutdown_connections 2023-07-14T19:03:55.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.796+0000 7f162ae55700 1 -- 172.21.15.78:0/2079856504 wait complete. 2023-07-14T19:03:55.812 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:03:55.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 -- 172.21.15.78:0/3633614129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c78101820 msgr2=0x7f2c781080e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/3633614129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c78101820 0x7f2c781080e0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f2c7000b0a0 tx=0x7f2c7000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 -- 172.21.15.78:0/3633614129 shutdown_connections 2023-07-14T19:03:55.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/3633614129 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c78101820 0x7f2c781080e0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.934 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/3633614129 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c780fdec0 0x7f2c780fe2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/3633614129 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c78100e90 0x7f2c781012e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 -- 172.21.15.78:0/3633614129 >> 172.21.15.78:0/3633614129 conn(0x7f2c780f98a0 msgr2=0x7f2c780fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 -- 172.21.15.78:0/3633614129 shutdown_connections 2023-07-14T19:03:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.930+0000 7f2c7e54f700 1 -- 172.21.15.78:0/3633614129 wait complete. 2023-07-14T19:03:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7e54f700 1 Processor -- start 2023-07-14T19:03:55.935 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7e54f700 1 -- start start 2023-07-14T19:03:55.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7e54f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c780fdec0 0x7f2c78100800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7e54f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c78100e90 0x7f2c780fee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7e54f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c78101820 0x7f2c780ff380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7e54f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c780ffb20 con 0x7f2c780fdec0 2023-07-14T19:03:55.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7e54f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c780ffc90 con 0x7f2c78101820 2023-07-14T19:03:55.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7e54f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c7819f970 con 0x7f2c78100e90 2023-07-14T19:03:55.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c77fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c780fdec0 0x7f2c78100800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c77fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c780fdec0 0x7f2c78100800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35712/0 (socket says 172.21.15.78:35712) 2023-07-14T19:03:55.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c77fff700 1 -- 172.21.15.78:0/733129871 learned_addr learned my addr 172.21.15.78:0/733129871 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:55.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c7caec700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c78101820 0x7f2c780ff380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c777fe700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c78100e90 0x7f2c780fee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c77fff700 1 -- 172.21.15.78:0/733129871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c78100e90 msgr2=0x7f2c780fee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c77fff700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c78100e90 0x7f2c780fee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c77fff700 1 -- 172.21.15.78:0/733129871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c78101820 msgr2=0x7f2c780ff380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:55.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c77fff700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c78101820 0x7f2c780ff380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:55.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.931+0000 7f2c77fff700 1 -- 172.21.15.78:0/733129871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c70009d00 con 0x7f2c780fdec0 2023-07-14T19:03:55.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.932+0000 7f2c77fff700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c780fdec0 0x7f2c78100800 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c970 tx=0x7f2c6800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:55.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.932+0000 7f2c757fa700 1 -- 172.21.15.78:0/733129871 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c68004220 con 0x7f2c780fdec0 2023-07-14T19:03:55.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.932+0000 7f2c757fa700 1 -- 172.21.15.78:0/733129871 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2c68007d30 con 0x7f2c780fdec0 2023-07-14T19:03:55.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.932+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c7819fc50 con 0x7f2c780fdec0 2023-07-14T19:03:55.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.932+0000 7f2c757fa700 1 -- 172.21.15.78:0/733129871 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c68003b80 con 0x7f2c780fdec0 2023-07-14T19:03:55.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.932+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c781a0160 con 0x7f2c780fdec0 2023-07-14T19:03:55.940 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.933+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c781071e0 con 0x7f2c780fdec0 2023-07-14T19:03:55.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.934+0000 7f2c757fa700 1 -- 172.21.15.78:0/733129871 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2c68003ce0 con 0x7f2c780fdec0 2023-07-14T19:03:55.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.935+0000 7f2c757fa700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c600a1220 0x7f2c600a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:55.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.935+0000 7f2c777fe700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c600a1220 0x7f2c600a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:55.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.935+0000 7f2c757fa700 1 -- 172.21.15.78:0/733129871 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f2c680ccd80 con 0x7f2c780fdec0 2023-07-14T19:03:55.941 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.935+0000 7f2c777fe700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c600a1220 0x7f2c600a36d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c005950 tx=0x7f2c6c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:55.942 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:55.938+0000 7f2c757fa700 1 -- 172.21.15.78:0/733129871 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2c68096060 con 0x7f2c780fdec0 2023-07-14T19:03:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.169+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c7802c8e0 con 0x7f2c780fdec0 2023-07-14T19:03:56.173 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.170+0000 7f2c757fa700 1 -- 172.21.15.78:0/733129871 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10808 (secure 0 0 0) 0x7f2c6801d090 con 0x7f2c780fdec0 2023-07-14T19:03:56.174 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:03:56.174 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:02:50.605732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24485":{"gid":24485,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:03:56.175 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c600a1220 msgr2=0x7f2c600a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c600a1220 0x7f2c600a36d0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2c6c005950 tx=0x7f2c6c0058e0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c780fdec0 msgr2=0x7f2c78100800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c780fdec0 0x7f2c78100800 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f2c6800c970 tx=0x7f2c6800cc80 comp rx=0 tx=0).stop 2023-07-14T19:03:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 shutdown_connections 2023-07-14T19:03:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2c600a1220 0x7f2c600a36d0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.176 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2c78101820 0x7f2c780ff380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2c780fdec0 0x7f2c78100800 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 --2- 172.21.15.78:0/733129871 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c78100e90 0x7f2c780fee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 >> 172.21.15.78:0/733129871 conn(0x7f2c780f98a0 msgr2=0x7f2c78104320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 shutdown_connections 2023-07-14T19:03:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.172+0000 7f2c7e54f700 1 -- 172.21.15.78:0/733129871 wait complete. 2023-07-14T19:03:56.177 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2023-07-14T19:03:56.188 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T19:02:30.340765+0000', 'modified': '2023-07-14T19:02:50.605732+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24503, 'mds_1': 24487, 'mds_2': 24485}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.102:6835/75062394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6834', 'nonce': 75062394}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 75062394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24464': {'gid': 24464, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.88:6835/3667124043', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3667124043}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 3667124043}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.102:6837/3933565644', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 3933565644}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 3933565644}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24485': {'gid': 24485, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.88:6837/1950772288', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6836', 'nonce': 1950772288}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 1950772288}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.102:6839/984471622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6838', 'nonce': 984471622}, {'type': 'v1', 'addr': '172.21.15.102:6839', 'nonce': 984471622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6839/1379790346', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 1379790346}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 1379790346}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T19:03:56.188 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-14T19:03:56.189 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T19:03:56.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 -- 172.21.15.78:0/564669680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce44100150 msgr2=0x7fce44100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 --2- 172.21.15.78:0/564669680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce44100150 0x7fce44100560 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fce38009990 tx=0x7fce38009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 -- 172.21.15.78:0/564669680 shutdown_connections 2023-07-14T19:03:56.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 --2- 172.21.15.78:0/564669680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce440ff810 0x7fce441070e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 --2- 172.21.15.78:0/564669680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce44100150 0x7fce44100560 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 --2- 172.21.15.78:0/564669680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce440fee80 0x7fce440ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 -- 172.21.15.78:0/564669680 >> 172.21.15.78:0/564669680 conn(0x7fce440f9980 msgr2=0x7fce440fbd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 -- 172.21.15.78:0/564669680 shutdown_connections 2023-07-14T19:03:56.312 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.307+0000 7fce4884c700 1 -- 172.21.15.78:0/564669680 wait complete. 2023-07-14T19:03:56.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4884c700 1 Processor -- start 2023-07-14T19:03:56.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4884c700 1 -- start start 2023-07-14T19:03:56.314 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4884c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce440fee80 0x7fce44102c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4884c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce440ff810 0x7fce44101270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4884c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce44100150 0x7fce441017b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4884c700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce44101f50 con 0x7fce440ff810 2023-07-14T19:03:56.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4884c700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce441020c0 con 0x7fce44100150 2023-07-14T19:03:56.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4884c700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4419f940 con 0x7fce440fee80 2023-07-14T19:03:56.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce440ff810 0x7fce44101270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.315 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce440ff810 0x7fce44101270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35728/0 (socket says 172.21.15.78:35728) 2023-07-14T19:03:56.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce41d9b700 1 -- 172.21.15.78:0/1577457715 learned_addr learned my addr 172.21.15.78:0/1577457715 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce42d9d700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce44100150 0x7fce441017b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce41d9b700 1 -- 172.21.15.78:0/1577457715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce440fee80 msgr2=0x7fce44102c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce4259c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce440fee80 0x7fce44102c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.308+0000 7fce41d9b700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce440fee80 0x7fce44102c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce41d9b700 1 -- 172.21.15.78:0/1577457715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce44100150 msgr2=0x7fce441017b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce41d9b700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce44100150 0x7fce441017b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce41d9b700 1 -- 172.21.15.78:0/1577457715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380095f0 con 0x7fce440ff810 2023-07-14T19:03:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce42d9d700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce44100150 0x7fce441017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce41d9b700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce440ff810 0x7fce44101270 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fce3400c920 tx=0x7fce3400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce337fe700 1 -- 172.21.15.78:0/1577457715 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34007b00 con 0x7fce440ff810 2023-07-14T19:03:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce337fe700 1 -- 172.21.15.78:0/1577457715 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fce34004cc0 con 0x7fce440ff810 2023-07-14T19:03:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce4884c700 1 -- 172.21.15.78:0/1577457715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4419fc20 con 0x7fce440ff810 2023-07-14T19:03:56.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce337fe700 1 -- 172.21.15.78:0/1577457715 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34005590 con 0x7fce440ff810 2023-07-14T19:03:56.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.309+0000 7fce4884c700 1 -- 172.21.15.78:0/1577457715 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce441a0130 con 0x7fce440ff810 2023-07-14T19:03:56.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.311+0000 7fce337fe700 1 -- 172.21.15.78:0/1577457715 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fce34004750 con 0x7fce440ff810 2023-07-14T19:03:56.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.312+0000 7fce337fe700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fce2c0a11e0 0x7fce2c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.312+0000 7fce4259c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fce2c0a11e0 0x7fce2c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.312+0000 7fce337fe700 1 -- 172.21.15.78:0/1577457715 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fce340ccdb0 con 0x7fce440ff810 2023-07-14T19:03:56.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.312+0000 7fce4259c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fce2c0a11e0 0x7fce2c0a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fce38005f50 tx=0x7fce38005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce549e4b80 1 Processor -- start 2023-07-14T19:03:56.323 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce549e4b80 1 -- start start 2023-07-14T19:03:56.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e2e320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e2ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2f460 0x5575f7e2f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce549e4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7d32c60 con 0x5575f7e2e860 2023-07-14T19:03:56.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce549e4b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7d859f0 con 0x5575f7e2f460 2023-07-14T19:03:56.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce549e4b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7d7ee80 con 0x5575f7e2df10 2023-07-14T19:03:56.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e2ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e2ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35736/0 (socket says 172.21.15.78:35736) 2023-07-14T19:03:56.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce41d9b700 1 -- 172.21.15.78:0/1507830926 learned_addr learned my addr 172.21.15.78:0/1507830926 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce42d9d700 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2f460 0x5575f7e2f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.313+0000 7fce4259c700 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e2e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce41d9b700 1 -- 172.21.15.78:0/1507830926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 msgr2=0x5575f7e2e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce41d9b700 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e2e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.325 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce41d9b700 1 -- 172.21.15.78:0/1507830926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2f460 msgr2=0x5575f7e2f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce41d9b700 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2f460 0x5575f7e2f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce41d9b700 1 -- 172.21.15.78:0/1507830926 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575f7bacd90 con 0x5575f7e2e860 2023-07-14T19:03:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce4259c700 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e2e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce42d9d700 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2f460 0x5575f7e2f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce41d9b700 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e2ee80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fce340a1c50 tx=0x7fce34099da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddc5579f96e5a839 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce30ff9700 1 -- 172.21.15.78:0/1507830926 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340980d0 con 0x5575f7e2e860 2023-07-14T19:03:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce30ff9700 1 -- 172.21.15.78:0/1507830926 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fce34007c60 con 0x5575f7e2e860 2023-07-14T19:03:56.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.314+0000 7fce30ff9700 1 -- 172.21.15.78:0/1507830926 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34004af0 con 0x5575f7e2e860 2023-07-14T19:03:56.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- 172.21.15.78:0/1507830926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 msgr2=0x5575f7e2ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e2ee80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fce340a1c50 tx=0x7fce34099da0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- 172.21.15.78:0/1507830926 shutdown_connections 2023-07-14T19:03:56.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2f460 0x5575f7e2f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e2e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 --2- 172.21.15.78:0/1507830926 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e2ee80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- 172.21.15.78:0/1507830926 >> 172.21.15.78:0/1507830926 conn(0x5575f7e23630 msgr2=0x5575f7e28980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- 172.21.15.78:0/1507830926 shutdown_connections 2023-07-14T19:03:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- 172.21.15.78:0/1507830926 wait complete. 2023-07-14T19:03:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- start start 2023-07-14T19:03:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 Processor -- start 2023-07-14T19:03:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e37970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2e860 0x5575f7e37eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e3eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7d1ea70 con 0x5575f7e2f460 2023-07-14T19:03:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7d1e770 con 0x5575f7e2e860 2023-07-14T19:03:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7c028d0 con 0x5575f7e2df10 2023-07-14T19:03:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce549e4b80 10 client.0 _ll_register_callbacks cb 0x5575f7d1c710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:03:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce42d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e3eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce42d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e3eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35744/0 (socket says 172.21.15.78:35744) 2023-07-14T19:03:56.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce42d9d700 1 -- 172.21.15.78:0/3778620040 learned_addr learned my addr 172.21.15.78:0/3778620040 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.315+0000 7fce4259c700 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e37970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce41d9b700 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2e860 0x5575f7e37eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce42d9d700 1 -- 172.21.15.78:0/3778620040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 msgr2=0x5575f7e37970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce42d9d700 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e37970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce42d9d700 1 -- 172.21.15.78:0/3778620040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2e860 msgr2=0x5575f7e37eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce42d9d700 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2e860 0x5575f7e37eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce42d9d700 1 -- 172.21.15.78:0/3778620040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce441a0130 con 0x5575f7e2f460 2023-07-14T19:03:56.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce42d9d700 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e3eef0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00ac40 tx=0x7fce3c00ac70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfff8eca45f3b82b server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce29ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:03:56.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce29ffb700 1 -- 172.21.15.78:0/3778620040 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c007bb0 con 0x5575f7e2f460 2023-07-14T19:03:56.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce29ffb700 1 -- 172.21.15.78:0/3778620040 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fce3c006e40 con 0x5575f7e2f460 2023-07-14T19:03:56.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce549e4b80 1 -- 172.21.15.78:0/3778620040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5575f7cd84a0 con 0x5575f7e2f460 2023-07-14T19:03:56.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce29ffb700 1 -- 172.21.15.78:0/3778620040 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce3c007d10 con 0x5575f7e2f460 2023-07-14T19:03:56.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce549e4b80 10 client.15228 fetch_fsmap learned FSMap version 23 2023-07-14T19:03:56.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce549e4b80 1 -- 172.21.15.78:0/3778620040 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5575f7b95c10 con 0x5575f7e2f460 2023-07-14T19:03:56.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce29ffb700 1 -- 172.21.15.78:0/3778620040 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5575f7cd84a0 con 0x5575f7e2f460 2023-07-14T19:03:56.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce29ffb700 1 -- 172.21.15.78:0/3778620040 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fce3c015390 con 0x5575f7e2f460 2023-07-14T19:03:56.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.316+0000 7fce549e4b80 10 client.15228 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:03:56.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.317+0000 7fce549e4b80 10 client.15228 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:03:56.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.317+0000 7fce549e4b80 20 client.15228 populate_metadata read hostname 'smithi078' 2023-07-14T19:03:56.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.317+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x5575f7e42bc0 0x5575f7e41fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.317+0000 7fce549e4b80 4 client.15228 mds_command: new command op to 24485 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:03:56.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.317+0000 7fce549e4b80 1 -- 172.21.15.78:0/3778620040 --> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5575f7dfcd50 con 0x5575f7e42bc0 2023-07-14T19:03:56.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.317+0000 7fce4259c700 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x5575f7e42bc0 0x5575f7e41fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.317+0000 7fce29ffb700 10 client.15228 ms_handle_connect on v2:172.21.15.88:6836/1950772288 2023-07-14T19:03:56.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.317+0000 7fce4259c700 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x5575f7e42bc0 0x5575f7e41fc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60566e6c137eda48 server_cookie=3b235db5312db6ed in_seq=0 out_seq=0 2023-07-14T19:03:56.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce29ffb700 1 -- 172.21.15.78:0/3778620040 <== mds.2 v2:172.21.15.88:6836/1950772288 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5575f7dfcd50 con 0x5575f7e42bc0 2023-07-14T19:03:56.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce29ffb700 10 client.15228 handle_command_reply: tid=0 2023-07-14T19:03:56.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 client.15228 shutdown 2023-07-14T19:03:56.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 -- 172.21.15.78:0/3778620040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 msgr2=0x5575f7e3eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e3eef0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fce3c00ac40 tx=0x7fce3c00ac70 comp rx=0 tx=0).stop 2023-07-14T19:03:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 -- 172.21.15.78:0/3778620040 shutdown_connections 2023-07-14T19:03:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e3eef0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2df10 0x5575f7e37970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x5575f7e42bc0 0x5575f7e41fc0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3778620040 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2e860 0x5575f7e37eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.334 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce29ffb700 0 client.15228 ms_handle_reset on v2:172.21.15.88:6836/1950772288 2023-07-14T19:03:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 -- 172.21.15.78:0/3778620040 >> 172.21.15.78:0/3778620040 conn(0x5575f7e23630 msgr2=0x5575f7e27600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 -- 172.21.15.78:0/3778620040 shutdown_connections 2023-07-14T19:03:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 1 -- 172.21.15.78:0/3778620040 wait complete. 2023-07-14T19:03:56.335 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.318+0000 7fce549e4b80 20 client.15228 trim_cache size 0 max 16384 2023-07-14T19:03:56.342 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.335+0000 7fce549e4b80 1 Processor -- start 2023-07-14T19:03:56.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce549e4b80 1 -- start start 2023-07-14T19:03:56.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e3f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e4b910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2f460 0x5575f7e41fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce549e4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7e29e30 con 0x5575f7e2e860 2023-07-14T19:03:56.343 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce549e4b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7d4d790 con 0x5575f7e2df10 2023-07-14T19:03:56.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce549e4b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7d3d480 con 0x5575f7e2f460 2023-07-14T19:03:56.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e4b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e4b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35750/0 (socket says 172.21.15.78:35750) 2023-07-14T19:03:56.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 -- 172.21.15.78:0/3146387378 learned_addr learned my addr 172.21.15.78:0/3146387378 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce4259c700 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e3f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 -- 172.21.15.78:0/3146387378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2f460 msgr2=0x5575f7e41fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.344 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce42d9d700 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2f460 0x5575f7e41fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2f460 0x5575f7e41fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 -- 172.21.15.78:0/3146387378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 msgr2=0x5575f7e3f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e3f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 -- 172.21.15.78:0/3146387378 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575f7e28f00 con 0x5575f7e2e860 2023-07-14T19:03:56.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce4259c700 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e3f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:56.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce41d9b700 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e4b910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fce34099a30 tx=0x7fce340ca8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=193c0662829e9b82 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce317fa700 1 -- 172.21.15.78:0/3146387378 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce340ca680 con 0x5575f7e2e860 2023-07-14T19:03:56.345 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce317fa700 1 -- 172.21.15.78:0/3146387378 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fce340980d0 con 0x5575f7e2e860 2023-07-14T19:03:56.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.336+0000 7fce317fa700 1 -- 172.21.15.78:0/3146387378 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3409a1b0 con 0x5575f7e2e860 2023-07-14T19:03:56.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 -- 172.21.15.78:0/3146387378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 msgr2=0x5575f7e4b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e4b910 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fce34099a30 tx=0x7fce340ca8b0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 -- 172.21.15.78:0/3146387378 shutdown_connections 2023-07-14T19:03:56.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e3f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2f460 0x5575f7e41fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.346 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 --2- 172.21.15.78:0/3146387378 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2e860 0x5575f7e4b910 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 -- 172.21.15.78:0/3146387378 >> 172.21.15.78:0/3146387378 conn(0x5575f7e23630 msgr2=0x5575f7e280a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 -- 172.21.15.78:0/3146387378 shutdown_connections 2023-07-14T19:03:56.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 -- 172.21.15.78:0/3146387378 wait complete. 2023-07-14T19:03:56.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 -- start start 2023-07-14T19:03:56.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.337+0000 7fce549e4b80 1 Processor -- start 2023-07-14T19:03:56.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e49900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2e860 0x5575f7e94810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e96d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7e972d0 con 0x5575f7e2f460 2023-07-14T19:03:56.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7e97440 con 0x5575f7e2df10 2023-07-14T19:03:56.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575f7e97600 con 0x5575f7e2e860 2023-07-14T19:03:56.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 10 client.0 _ll_register_callbacks cb 0x7fce3c00a490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:03:56.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e96d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e96d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35758/0 (socket says 172.21.15.78:35758) 2023-07-14T19:03:56.348 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 -- 172.21.15.78:0/4279645348 learned_addr learned my addr 172.21.15.78:0/4279645348 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce41d9b700 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2e860 0x5575f7e94810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce4259c700 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e49900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 -- 172.21.15.78:0/4279645348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2e860 msgr2=0x5575f7e94810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2e860 0x5575f7e94810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 -- 172.21.15.78:0/4279645348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 msgr2=0x5575f7e49900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e49900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.349 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 -- 172.21.15.78:0/4279645348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5575f7bacd90 con 0x5575f7e2f460 2023-07-14T19:03:56.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce42d9d700 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e96d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0157f0 tx=0x7fce3c007070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=382499b5f53d4d7f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce29ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:03:56.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce29ffb700 1 -- 172.21.15.78:0/4279645348 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3c00f070 con 0x5575f7e2f460 2023-07-14T19:03:56.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce29ffb700 1 -- 172.21.15.78:0/4279645348 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fce3c0079b0 con 0x5575f7e2f460 2023-07-14T19:03:56.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 1 -- 172.21.15.78:0/4279645348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce34007c60 con 0x5575f7e2f460 2023-07-14T19:03:56.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce29ffb700 1 -- 172.21.15.78:0/4279645348 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce3c015b20 con 0x5575f7e2f460 2023-07-14T19:03:56.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 10 client.15234 fetch_fsmap learned FSMap version 23 2023-07-14T19:03:56.350 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.338+0000 7fce549e4b80 1 -- 172.21.15.78:0/4279645348 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5575f7e3f0e0 con 0x5575f7e2f460 2023-07-14T19:03:56.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce29ffb700 1 -- 172.21.15.78:0/4279645348 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fce34007c60 con 0x5575f7e2f460 2023-07-14T19:03:56.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce29ffb700 1 -- 172.21.15.78:0/4279645348 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fce3c008040 con 0x5575f7e2f460 2023-07-14T19:03:56.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce549e4b80 10 client.15234 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:03:56.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce549e4b80 10 client.15234 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:03:56.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce549e4b80 20 client.15234 populate_metadata read hostname 'smithi078' 2023-07-14T19:03:56.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce549e4b80 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x5575f7e9c2d0 0x5575f7e98c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.351 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce549e4b80 4 client.15234 mds_command: new command op to 24485 tid=0[{"prefix": "status"}] 2023-07-14T19:03:56.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce549e4b80 1 -- 172.21.15.78:0/4279645348 --> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5575f7d77e60 con 0x5575f7e9c2d0 2023-07-14T19:03:56.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce4259c700 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x5575f7e9c2d0 0x5575f7e98c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce29ffb700 10 client.15234 ms_handle_connect on v2:172.21.15.88:6836/1950772288 2023-07-14T19:03:56.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.339+0000 7fce4259c700 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x5575f7e9c2d0 0x5575f7e98c40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9f3f702e7ec0a7d5 server_cookie=59634e45a7c3d486 in_seq=0 out_seq=0 2023-07-14T19:03:56.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce29ffb700 1 -- 172.21.15.78:0/4279645348 <== mds.2 v2:172.21.15.88:6836/1950772288 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5575f7d77e60 con 0x5575f7e9c2d0 2023-07-14T19:03:56.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce29ffb700 10 client.15234 handle_command_reply: tid=0 2023-07-14T19:03:56.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 client.15234 shutdown 2023-07-14T19:03:56.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 -- 172.21.15.78:0/4279645348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 msgr2=0x5575f7e96d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e96d60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fce3c0157f0 tx=0x7fce3c007070 comp rx=0 tx=0).stop 2023-07-14T19:03:56.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 -- 172.21.15.78:0/4279645348 shutdown_connections 2023-07-14T19:03:56.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x5575f7e9c2d0 0x5575f7e98c40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5575f7e2df10 0x5575f7e49900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5575f7e2f460 0x5575f7e96d60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce29ffb700 0 client.15234 ms_handle_reset on v2:172.21.15.88:6836/1950772288 2023-07-14T19:03:56.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 --2- 172.21.15.78:0/4279645348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5575f7e2e860 0x5575f7e94810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 -- 172.21.15.78:0/4279645348 >> 172.21.15.78:0/4279645348 conn(0x5575f7e23630 msgr2=0x5575f7e41fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 -- 172.21.15.78:0/4279645348 shutdown_connections 2023-07-14T19:03:56.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 1 -- 172.21.15.78:0/4279645348 wait complete. 2023-07-14T19:03:56.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce549e4b80 20 client.15234 trim_cache size 0 max 16384 2023-07-14T19:03:56.355 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:03:56.356 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T19:03:56.356 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2023-07-14T19:03:56.356 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24485, 2023-07-14T19:03:56.356 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-07-14T19:03:56.356 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-07-14T19:03:56.356 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-07-14T19:03:56.357 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 66.140992839000006, 2023-07-14T19:03:56.357 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2023-07-14T19:03:56.357 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2023-07-14T19:03:56.357 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T19:03:56.357 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 100.227236847 2023-07-14T19:03:56.357 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:03:56.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 -- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fce2c0a11e0 msgr2=0x7fce2c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fce2c0a11e0 0x7fce2c0a3690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fce38005f50 tx=0x7fce38005e80 comp rx=0 tx=0).stop 2023-07-14T19:03:56.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 -- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce440ff810 msgr2=0x7fce44101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce440ff810 0x7fce44101270 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fce3400c920 tx=0x7fce3400cce0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 -- 172.21.15.78:0/1577457715 shutdown_connections 2023-07-14T19:03:56.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fce2c0a11e0 0x7fce2c0a3690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fce44100150 0x7fce441017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce440fee80 0x7fce44102c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 --2- 172.21.15.78:0/1577457715 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fce440ff810 0x7fce44101270 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.340+0000 7fce4884c700 1 -- 172.21.15.78:0/1577457715 >> 172.21.15.78:0/1577457715 conn(0x7fce440f9980 msgr2=0x7fce440fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.341+0000 7fce4884c700 1 -- 172.21.15.78:0/1577457715 shutdown_connections 2023-07-14T19:03:56.360 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.341+0000 7fce4884c700 1 -- 172.21.15.78:0/1577457715 wait complete. 2023-07-14T19:03:56.361 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T19:03:56.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:56 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/918068847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:56 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2079856504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:56 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/733129871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.479+0000 7f7d04142700 1 -- 172.21.15.78:0/4131669080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 msgr2=0x7f7cfc0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.479+0000 7f7d04142700 1 --2- 172.21.15.78:0/4131669080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 0x7f7cfc0fef20 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009990 tx=0x7f7cf8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.479+0000 7f7d04142700 1 -- 172.21.15.78:0/4131669080 shutdown_connections 2023-07-14T19:03:56.485 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.479+0000 7f7d04142700 1 --2- 172.21.15.78:0/4131669080 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cfc100690 0x7f7cfc106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.479+0000 7f7d04142700 1 --2- 172.21.15.78:0/4131669080 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 0x7f7cfc0fef20 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.479+0000 7f7d04142700 1 --2- 172.21.15.78:0/4131669080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cfc0ffd00 0x7f7cfc100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.479+0000 7f7d04142700 1 -- 172.21.15.78:0/4131669080 >> 172.21.15.78:0/4131669080 conn(0x7f7cfc0fa130 msgr2=0x7f7cfc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 -- 172.21.15.78:0/4131669080 shutdown_connections 2023-07-14T19:03:56.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 -- 172.21.15.78:0/4131669080 wait complete. 2023-07-14T19:03:56.486 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 Processor -- start 2023-07-14T19:03:56.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 -- start start 2023-07-14T19:03:56.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 0x7f7cfc1958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cfc0ffd00 0x7f7cfc195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cfc100690 0x7f7cfc19cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc19d470 con 0x7f7cfc0feb10 2023-07-14T19:03:56.487 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc19d5b0 con 0x7f7cfc0ffd00 2023-07-14T19:03:56.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d04142700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc19d740 con 0x7f7cfc100690 2023-07-14T19:03:56.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d01ede700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 0x7f7cfc1958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d01ede700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 0x7f7cfc1958f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35768/0 (socket says 172.21.15.78:35768) 2023-07-14T19:03:56.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.480+0000 7f7d01ede700 1 -- 172.21.15.78:0/2951147680 learned_addr learned my addr 172.21.15.78:0/2951147680 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d026df700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cfc100690 0x7f7cfc19cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d01ede700 1 -- 172.21.15.78:0/2951147680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cfc100690 msgr2=0x7f7cfc19cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.488 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d016dd700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cfc0ffd00 0x7f7cfc195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d01ede700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cfc100690 0x7f7cfc19cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d01ede700 1 -- 172.21.15.78:0/2951147680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cfc0ffd00 msgr2=0x7f7cfc195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d01ede700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cfc0ffd00 0x7f7cfc195e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d01ede700 1 -- 172.21.15.78:0/2951147680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf80095f0 con 0x7f7cfc0feb10 2023-07-14T19:03:56.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d01ede700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 0x7f7cfc1958f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8000c00 tx=0x7f7cf80040b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2951147680 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf801d070 con 0x7f7cfc0feb10 2023-07-14T19:03:56.489 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2951147680 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7cf80041d0 con 0x7f7cfc0feb10 2023-07-14T19:03:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d04142700 1 -- 172.21.15.78:0/2951147680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cfc19d9c0 con 0x7f7cfc0feb10 2023-07-14T19:03:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2951147680 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf800f6c0 con 0x7f7cfc0feb10 2023-07-14T19:03:56.490 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.481+0000 7f7d04142700 1 -- 172.21.15.78:0/2951147680 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc19dde0 con 0x7f7cfc0feb10 2023-07-14T19:03:56.494 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.483+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2951147680 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7cf8022470 con 0x7f7cfc0feb10 2023-07-14T19:03:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.483+0000 7f7cf2ffd700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ce80a11e0 0x7f7ce80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.484+0000 7f7cf2ffd700 1 -- 172.21.15.78:0/2951147680 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7cf80ce5d0 con 0x7f7cfc0feb10 2023-07-14T19:03:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.484+0000 7f7d016dd700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ce80a11e0 0x7f7ce80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.484+0000 7f7d016dd700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ce80a11e0 0x7f7ce80a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7cec005950 tx=0x7f7cec0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d102dab80 1 Processor -- start 2023-07-14T19:03:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d102dab80 1 -- start start 2023-07-14T19:03:56.495 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 0x5623fb751320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 0x5623fb751e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb7528d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d102dab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb655c60 con 0x5623fb752460 2023-07-14T19:03:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d102dab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb6a89f0 con 0x5623fb750f10 2023-07-14T19:03:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d102dab80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb6a1e80 con 0x5623fb751860 2023-07-14T19:03:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d026df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb7528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.496 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d026df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb7528d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35780/0 (socket says 172.21.15.78:35780) 2023-07-14T19:03:56.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d026df700 1 -- 172.21.15.78:0/4029827070 learned_addr learned my addr 172.21.15.78:0/4029827070 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.485+0000 7f7d01ede700 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 0x5623fb751320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7d016dd700 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 0x5623fb751e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7d026df700 1 -- 172.21.15.78:0/4029827070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 msgr2=0x5623fb751e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7d026df700 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 0x5623fb751e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7d026df700 1 -- 172.21.15.78:0/4029827070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 msgr2=0x5623fb751320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.497 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7d026df700 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 0x5623fb751320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7d026df700 1 -- 172.21.15.78:0/4029827070 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623fb4cfd90 con 0x5623fb752460 2023-07-14T19:03:56.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7d026df700 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb7528d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009440 tx=0x7f7cf4009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a6f34462be59d0f server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7ce7fff700 1 -- 172.21.15.78:0/4029827070 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf400e070 con 0x5623fb752460 2023-07-14T19:03:56.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7ce7fff700 1 -- 172.21.15.78:0/4029827070 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7cf400f040 con 0x5623fb752460 2023-07-14T19:03:56.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.486+0000 7f7ce7fff700 1 -- 172.21.15.78:0/4029827070 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4015a10 con 0x5623fb752460 2023-07-14T19:03:56.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 -- 172.21.15.78:0/4029827070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 msgr2=0x5623fb7528d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb7528d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4009440 tx=0x7f7cf4009920 comp rx=0 tx=0).stop 2023-07-14T19:03:56.498 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 -- 172.21.15.78:0/4029827070 shutdown_connections 2023-07-14T19:03:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 0x5623fb751320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb7528d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 --2- 172.21.15.78:0/4029827070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 0x5623fb751e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 -- 172.21.15.78:0/4029827070 >> 172.21.15.78:0/4029827070 conn(0x5623fb746630 msgr2=0x5623fb74b980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 -- 172.21.15.78:0/4029827070 shutdown_connections 2023-07-14T19:03:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 -- 172.21.15.78:0/4029827070 wait complete. 2023-07-14T19:03:56.499 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 -- start start 2023-07-14T19:03:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 Processor -- start 2023-07-14T19:03:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 0x5623fb75a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 0x5623fb75ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.487+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb761e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d102dab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb641a70 con 0x5623fb752460 2023-07-14T19:03:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d102dab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb641770 con 0x5623fb750f10 2023-07-14T19:03:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d102dab80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb5258d0 con 0x5623fb751860 2023-07-14T19:03:56.500 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d102dab80 10 client.0 _ll_register_callbacks cb 0x5623fb63f710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:03:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb761e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb761e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35786/0 (socket says 172.21.15.78:35786) 2023-07-14T19:03:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 -- 172.21.15.78:0/1133724409 learned_addr learned my addr 172.21.15.78:0/1133724409 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d016dd700 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 0x5623fb75ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 -- 172.21.15.78:0/1133724409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 msgr2=0x5623fb75ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d01ede700 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 0x5623fb75a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.501 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 0x5623fb75ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 -- 172.21.15.78:0/1133724409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 msgr2=0x5623fb75a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 0x5623fb75a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 -- 172.21.15.78:0/1133724409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf40090f0 con 0x5623fb752460 2023-07-14T19:03:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d026df700 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb761e70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4016a00 tx=0x7f7cf4016ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd7e02cdc1e00fb4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7ce57fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:03:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1133724409 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf400e040 con 0x5623fb752460 2023-07-14T19:03:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1133724409 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7cf4013750 con 0x5623fb752460 2023-07-14T19:03:56.502 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d102dab80 1 -- 172.21.15.78:0/1133724409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5623fb5fb4a0 con 0x5623fb752460 2023-07-14T19:03:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1133724409 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cf4013910 con 0x5623fb752460 2023-07-14T19:03:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d102dab80 10 client.15246 fetch_fsmap learned FSMap version 23 2023-07-14T19:03:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7d102dab80 1 -- 172.21.15.78:0/1133724409 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5623fb4b8c10 con 0x5623fb752460 2023-07-14T19:03:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.488+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1133724409 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x5623fb5fb4a0 con 0x5623fb752460 2023-07-14T19:03:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1133724409 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7cf401c050 con 0x5623fb752460 2023-07-14T19:03:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7d102dab80 10 client.15246 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:03:56.503 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7d102dab80 10 client.15246 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:03:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7d102dab80 20 client.15246 populate_metadata read hostname 'smithi078' 2023-07-14T19:03:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x5623fb765ab0 0x5623fb764ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7d102dab80 4 client.15246 mds_command: new command op to 24487 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:03:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7d102dab80 1 -- 172.21.15.78:0/1133724409 --> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5623fb71fd50 con 0x5623fb765ab0 2023-07-14T19:03:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7d01ede700 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x5623fb765ab0 0x5623fb764ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7ce57fa700 10 client.15246 ms_handle_connect on v2:172.21.15.102:6838/984471622 2023-07-14T19:03:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.489+0000 7f7d01ede700 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x5623fb765ab0 0x5623fb764ec0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9d430440408ea900 server_cookie=ca876d5058192b01 in_seq=0 out_seq=0 2023-07-14T19:03:56.504 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1133724409 <== mds.1 v2:172.21.15.102:6838/984471622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x5623fb71fd50 con 0x5623fb765ab0 2023-07-14T19:03:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7ce57fa700 10 client.15246 handle_command_reply: tid=0 2023-07-14T19:03:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 client.15246 shutdown 2023-07-14T19:03:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 -- 172.21.15.78:0/1133724409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 msgr2=0x5623fb761e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb761e70 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4016a00 tx=0x7f7cf4016ae0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 -- 172.21.15.78:0/1133724409 shutdown_connections 2023-07-14T19:03:56.505 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x5623fb765ab0 0x5623fb764ec0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.506 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb750f10 0x5623fb75a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb752460 0x5623fb761e70 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1133724409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb751860 0x5623fb75ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7ce57fa700 0 client.15246 ms_handle_reset on v2:172.21.15.102:6838/984471622 2023-07-14T19:03:56.507 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 -- 172.21.15.78:0/1133724409 >> 172.21.15.78:0/1133724409 conn(0x5623fb746630 msgr2=0x5623fb74a600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.490+0000 7f7d102dab80 1 -- 172.21.15.78:0/1133724409 shutdown_connections 2023-07-14T19:03:56.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.491+0000 7f7d102dab80 1 -- 172.21.15.78:0/1133724409 wait complete. 2023-07-14T19:03:56.508 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.491+0000 7f7d102dab80 20 client.15246 trim_cache size 0 max 16384 2023-07-14T19:03:56.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d102dab80 1 Processor -- start 2023-07-14T19:03:56.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d102dab80 1 -- start start 2023-07-14T19:03:56.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb755340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb755880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 0x5623fb75a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.513 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d102dab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb531de0 con 0x5623fb751860 2023-07-14T19:03:56.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d102dab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb660480 con 0x5623fb752460 2023-07-14T19:03:56.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d102dab80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb63dd50 con 0x5623fb750f10 2023-07-14T19:03:56.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb755880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb755880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35792/0 (socket says 172.21.15.78:35792) 2023-07-14T19:03:56.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 -- 172.21.15.78:0/3837049505 learned_addr learned my addr 172.21.15.78:0/3837049505 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d01ede700 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb755340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.514 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 -- 172.21.15.78:0/3837049505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 msgr2=0x5623fb755340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb755340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d026df700 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 0x5623fb75a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 -- 172.21.15.78:0/3837049505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 msgr2=0x5623fb75a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 0x5623fb75a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 -- 172.21.15.78:0/3837049505 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623fb4f6200 con 0x5623fb751860 2023-07-14T19:03:56.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7d016dd700 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb755880 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00ecd0 tx=0x7f7cec011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb55d5a0f325ac1 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.515 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.508+0000 7f7cf0ff9700 1 -- 172.21.15.78:0/3837049505 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec00e7f0 con 0x5623fb751860 2023-07-14T19:03:56.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7cf0ff9700 1 -- 172.21.15.78:0/3837049505 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7cec011bb0 con 0x5623fb751860 2023-07-14T19:03:56.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7cf0ff9700 1 -- 172.21.15.78:0/3837049505 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec01f650 con 0x5623fb751860 2023-07-14T19:03:56.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 -- 172.21.15.78:0/3837049505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 msgr2=0x5623fb755880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb755880 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00ecd0 tx=0x7f7cec011000 comp rx=0 tx=0).stop 2023-07-14T19:03:56.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 -- 172.21.15.78:0/3837049505 shutdown_connections 2023-07-14T19:03:56.516 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 0x5623fb75a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb755340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 --2- 172.21.15.78:0/3837049505 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb755880 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 -- 172.21.15.78:0/3837049505 >> 172.21.15.78:0/3837049505 conn(0x5623fb746630 msgr2=0x5623fb7587e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 -- 172.21.15.78:0/3837049505 shutdown_connections 2023-07-14T19:03:56.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 -- 172.21.15.78:0/3837049505 wait complete. 2023-07-14T19:03:56.517 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 -- start start 2023-07-14T19:03:56.518 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.509+0000 7f7d102dab80 1 Processor -- start 2023-07-14T19:03:56.520 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb7b9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb7bbc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d102dab80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 0x5623fb7be270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d102dab80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb7be9d0 con 0x5623fb751860 2023-07-14T19:03:56.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d102dab80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb7beb40 con 0x5623fb752460 2023-07-14T19:03:56.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d102dab80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623fb7bed00 con 0x5623fb750f10 2023-07-14T19:03:56.521 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d102dab80 10 client.0 _ll_register_callbacks cb 0x7f7cf8097220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:03:56.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb7bbc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb7bbc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35798/0 (socket says 172.21.15.78:35798) 2023-07-14T19:03:56.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 -- 172.21.15.78:0/1987159493 learned_addr learned my addr 172.21.15.78:0/1987159493 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d026df700 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 0x5623fb7be270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d01ede700 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb7b9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 -- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 msgr2=0x5623fb7b9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb7b9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.522 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 -- 172.21.15.78:0/1987159493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 msgr2=0x5623fb7be270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 0x5623fb7be270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 -- 172.21.15.78:0/1987159493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cec00e950 con 0x5623fb751860 2023-07-14T19:03:56.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d01ede700 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb7b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:56.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7d016dd700 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb7bbc80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00e5d0 tx=0x7f7cec011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488775c1c86b5f6d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7ce57fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:03:56.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.510+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1987159493 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cec02c070 con 0x5623fb751860 2023-07-14T19:03:56.523 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1987159493 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7cec005250 con 0x5623fb751860 2023-07-14T19:03:56.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 1 -- 172.21.15.78:0/1987159493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7cf400f040 con 0x5623fb751860 2023-07-14T19:03:56.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1987159493 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7cec005410 con 0x5623fb751860 2023-07-14T19:03:56.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 10 client.15252 fetch_fsmap learned FSMap version 23 2023-07-14T19:03:56.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 1 -- 172.21.15.78:0/1987159493 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x5623fb4ce260 con 0x5623fb751860 2023-07-14T19:03:56.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1987159493 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7cf400f040 con 0x5623fb751860 2023-07-14T19:03:56.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1987159493 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f7cec025020 con 0x5623fb751860 2023-07-14T19:03:56.524 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 10 client.15252 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:03:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 10 client.15252 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:03:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 20 client.15252 populate_metadata read hostname 'smithi078' 2023-07-14T19:03:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x5623fb7c3840 0x5623fb7c01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 4 client.15252 mds_command: new command op to 24487 tid=0[{"prefix": "status"}] 2023-07-14T19:03:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d102dab80 1 -- 172.21.15.78:0/1987159493 --> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5623fb69ae60 con 0x5623fb7c3840 2023-07-14T19:03:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.511+0000 7f7d01ede700 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x5623fb7c3840 0x5623fb7c01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7ce57fa700 10 client.15252 ms_handle_connect on v2:172.21.15.102:6838/984471622 2023-07-14T19:03:56.525 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d01ede700 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x5623fb7c3840 0x5623fb7c01d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=981c3affea5c7592 server_cookie=fb67d08ecbbf4a5a in_seq=0 out_seq=0 2023-07-14T19:03:56.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7ce57fa700 1 -- 172.21.15.78:0/1987159493 <== mds.1 v2:172.21.15.102:6838/984471622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5623fb69ae60 con 0x5623fb7c3840 2023-07-14T19:03:56.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7ce57fa700 10 client.15252 handle_command_reply: tid=0 2023-07-14T19:03:56.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 client.15252 shutdown 2023-07-14T19:03:56.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 -- 172.21.15.78:0/1987159493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 msgr2=0x5623fb7bbc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb7bbc80 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f7cec00e5d0 tx=0x7f7cec011f50 comp rx=0 tx=0).stop 2023-07-14T19:03:56.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 -- 172.21.15.78:0/1987159493 shutdown_connections 2023-07-14T19:03:56.526 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x5623fb7c3840 0x5623fb7c01d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x5623fb752460 0x5623fb7be270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623fb750f10 0x5623fb7b9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7ce57fa700 0 client.15252 ms_handle_reset on v2:172.21.15.102:6838/984471622 2023-07-14T19:03:56.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 --2- 172.21.15.78:0/1987159493 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x5623fb751860 0x5623fb7bbc80 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 -- 172.21.15.78:0/1987159493 >> 172.21.15.78:0/1987159493 conn(0x5623fb746630 msgr2=0x5623fb755340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 -- 172.21.15.78:0/1987159493 shutdown_connections 2023-07-14T19:03:56.527 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 1 -- 172.21.15.78:0/1987159493 wait complete. 2023-07-14T19:03:56.528 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.512+0000 7f7d102dab80 20 client.15252 trim_cache size 0 max 16384 2023-07-14T19:03:56.528 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:03:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T19:03:56.528 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2023-07-14T19:03:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24487, 2023-07-14T19:03:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-07-14T19:03:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-07-14T19:03:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-07-14T19:03:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 67.326195312999999, 2023-07-14T19:03:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2023-07-14T19:03:56.529 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2023-07-14T19:03:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T19:03:56.530 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 98.255871377000005 2023-07-14T19:03:56.530 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:03:56.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 -- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ce80a11e0 msgr2=0x7f7ce80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ce80a11e0 0x7f7ce80a3690 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7cec005950 tx=0x7f7cec0058e0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 -- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 msgr2=0x7f7cfc1958f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 0x7f7cfc1958f0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8000c00 tx=0x7f7cf80040b0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 -- 172.21.15.78:0/2951147680 shutdown_connections 2023-07-14T19:03:56.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7ce80a11e0 0x7f7ce80a3690 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.531 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7cfc100690 0x7f7cfc19cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7cfc0feb10 0x7f7cfc1958f0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 --2- 172.21.15.78:0/2951147680 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7cfc0ffd00 0x7f7cfc195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 -- 172.21.15.78:0/2951147680 >> 172.21.15.78:0/2951147680 conn(0x7f7cfc0fa130 msgr2=0x7f7cfc0677e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 -- 172.21.15.78:0/2951147680 shutdown_connections 2023-07-14T19:03:56.532 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.513+0000 7f7d04142700 1 -- 172.21.15.78:0/2951147680 wait complete. 2023-07-14T19:03:56.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:56 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/918068847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:56 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2079856504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.533 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:56 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/733129871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.534 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:03:56.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:56 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/918068847' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:56 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2079856504' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:56 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/733129871' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:03:56.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 -- 172.21.15.78:0/1437123918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed700ffc10 msgr2=0x7fed70100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 --2- 172.21.15.78:0/1437123918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed700ffc10 0x7fed70100020 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fed60009990 tx=0x7fed60009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 -- 172.21.15.78:0/1437123918 shutdown_connections 2023-07-14T19:03:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 --2- 172.21.15.78:0/1437123918 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed70101790 0x7fed70108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 --2- 172.21.15.78:0/1437123918 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed700ffc10 0x7fed70100020 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 --2- 172.21.15.78:0/1437123918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed70100e00 0x7fed70101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 -- 172.21.15.78:0/1437123918 >> 172.21.15.78:0/1437123918 conn(0x7fed700fb250 msgr2=0x7fed700fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 -- 172.21.15.78:0/1437123918 shutdown_connections 2023-07-14T19:03:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.653+0000 7fed77aac700 1 -- 172.21.15.78:0/1437123918 wait complete. 2023-07-14T19:03:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed77aac700 1 Processor -- start 2023-07-14T19:03:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed77aac700 1 -- start start 2023-07-14T19:03:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed77aac700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed700ffc10 0x7fed70195880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed77aac700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed70100e00 0x7fed70195dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed77aac700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed70101790 0x7fed7019ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed77aac700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7019d400 con 0x7fed70101790 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed77aac700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7019d540 con 0x7fed700ffc10 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed77aac700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7019d6d0 con 0x7fed70100e00 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed76049700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed70101790 0x7fed7019ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed76049700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed70101790 0x7fed7019ce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35814/0 (socket says 172.21.15.78:35814) 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed76049700 1 -- 172.21.15.78:0/4183247956 learned_addr learned my addr 172.21.15.78:0/4183247956 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed75848700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed700ffc10 0x7fed70195880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.654+0000 7fed75047700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed70100e00 0x7fed70195dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed76049700 1 -- 172.21.15.78:0/4183247956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed70100e00 msgr2=0x7fed70195dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed76049700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed70100e00 0x7fed70195dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed76049700 1 -- 172.21.15.78:0/4183247956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed700ffc10 msgr2=0x7fed70195880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed76049700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed700ffc10 0x7fed70195880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed76049700 1 -- 172.21.15.78:0/4183247956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed600095f0 con 0x7fed70101790 2023-07-14T19:03:56.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed76049700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed70101790 0x7fed7019ce30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fed68007d70 tx=0x7fed6800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed66ffd700 1 -- 172.21.15.78:0/4183247956 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6800ad40 con 0x7fed70101790 2023-07-14T19:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed66ffd700 1 -- 172.21.15.78:0/4183247956 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed680128b0 con 0x7fed70101790 2023-07-14T19:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed77aac700 1 -- 172.21.15.78:0/4183247956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7019d9b0 con 0x7fed70101790 2023-07-14T19:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed66ffd700 1 -- 172.21.15.78:0/4183247956 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed680115a0 con 0x7fed70101790 2023-07-14T19:03:56.663 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.655+0000 7fed77aac700 1 -- 172.21.15.78:0/4183247956 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7019dec0 con 0x7fed70101790 2023-07-14T19:03:56.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.657+0000 7fed66ffd700 1 -- 172.21.15.78:0/4183247956 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fed68012a20 con 0x7fed70101790 2023-07-14T19:03:56.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.658+0000 7fed66ffd700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed5c0a11e0 0x7fed5c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.658+0000 7fed75848700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed5c0a11e0 0x7fed5c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.658+0000 7fed66ffd700 1 -- 172.21.15.78:0/4183247956 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fed680ccea0 con 0x7fed70101790 2023-07-14T19:03:56.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.658+0000 7fed75848700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed5c0a11e0 0x7fed5c0a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fed6000c010 tx=0x7fed60005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed83c44b80 1 Processor -- start 2023-07-14T19:03:56.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed83c44b80 1 -- start start 2023-07-14T19:03:56.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a0320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a18d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed83c44b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665efa4c60 con 0x55665f0a0860 2023-07-14T19:03:56.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed83c44b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665eff79f0 con 0x55665f0a1460 2023-07-14T19:03:56.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed83c44b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665eff0e80 con 0x55665f09ff10 2023-07-14T19:03:56.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75047700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75047700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35820/0 (socket says 172.21.15.78:35820) 2023-07-14T19:03:56.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75047700 1 -- 172.21.15.78:0/1153090647 learned_addr learned my addr 172.21.15.78:0/1153090647 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed76049700 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75848700 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75047700 1 -- 172.21.15.78:0/1153090647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 msgr2=0x55665f0a0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75047700 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a0320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75047700 1 -- 172.21.15.78:0/1153090647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 msgr2=0x55665f0a18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75047700 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.659+0000 7fed75047700 1 -- 172.21.15.78:0/1153090647 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55665ee1ed90 con 0x55665f0a0860 2023-07-14T19:03:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed75047700 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a0e80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00bb10 tx=0x7fed6c00be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af04df131591f535 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed5bfff700 1 -- 172.21.15.78:0/1153090647 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c00dc40 con 0x55665f0a0860 2023-07-14T19:03:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed5bfff700 1 -- 172.21.15.78:0/1153090647 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed6c00dda0 con 0x55665f0a0860 2023-07-14T19:03:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed5bfff700 1 -- 172.21.15.78:0/1153090647 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c00f680 con 0x55665f0a0860 2023-07-14T19:03:56.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed83c44b80 1 -- 172.21.15.78:0/1153090647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 msgr2=0x55665f0a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed83c44b80 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a0e80 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00bb10 tx=0x7fed6c00be20 comp rx=0 tx=0).stop 2023-07-14T19:03:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed83c44b80 1 -- 172.21.15.78:0/1153090647 shutdown_connections 2023-07-14T19:03:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed83c44b80 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed83c44b80 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed83c44b80 1 --2- 172.21.15.78:0/1153090647 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a0e80 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.660+0000 7fed83c44b80 1 -- 172.21.15.78:0/1153090647 >> 172.21.15.78:0/1153090647 conn(0x55665f095630 msgr2=0x55665f09a980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 -- 172.21.15.78:0/1153090647 shutdown_connections 2023-07-14T19:03:56.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 -- 172.21.15.78:0/1153090647 wait complete. 2023-07-14T19:03:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 -- start start 2023-07-14T19:03:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 Processor -- start 2023-07-14T19:03:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f0a98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f0a9df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f0b0e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665ef90a70 con 0x55665f09ff10 2023-07-14T19:03:56.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665ef90770 con 0x55665f0a0860 2023-07-14T19:03:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665ee748d0 con 0x55665f0a1460 2023-07-14T19:03:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed83c44b80 10 client.0 _ll_register_callbacks cb 0x55665ef8e710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:03:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed75848700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f0a98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed75848700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f0a98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35824/0 (socket says 172.21.15.78:35824) 2023-07-14T19:03:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.661+0000 7fed75848700 1 -- 172.21.15.78:0/50777555 learned_addr learned my addr 172.21.15.78:0/50777555 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed75047700 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f0a9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed76049700 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f0b0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.674 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed75848700 1 -- 172.21.15.78:0/50777555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 msgr2=0x55665f0b0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed75848700 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f0b0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed75848700 1 -- 172.21.15.78:0/50777555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 msgr2=0x55665f0a9df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed75848700 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f0a9df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed75848700 1 -- 172.21.15.78:0/50777555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6c00b770 con 0x55665f09ff10 2023-07-14T19:03:56.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed75848700 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f0a98b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fed60015000 tx=0x7fed60009ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659fc8bb23e9678a server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed597fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:03:56.675 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed597fa700 1 -- 172.21.15.78:0/50777555 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed60004d70 con 0x55665f09ff10 2023-07-14T19:03:56.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed597fa700 1 -- 172.21.15.78:0/50777555 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fed60003820 con 0x55665f09ff10 2023-07-14T19:03:56.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed83c44b80 1 -- 172.21.15.78:0/50777555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55665ef4a4a0 con 0x55665f09ff10 2023-07-14T19:03:56.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed597fa700 1 -- 172.21.15.78:0/50777555 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed600039e0 con 0x55665f09ff10 2023-07-14T19:03:56.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed83c44b80 10 client.15264 fetch_fsmap learned FSMap version 23 2023-07-14T19:03:56.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed83c44b80 1 -- 172.21.15.78:0/50777555 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55665ee07c10 con 0x55665f09ff10 2023-07-14T19:03:56.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed597fa700 1 -- 172.21.15.78:0/50777555 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55665ef4a4a0 con 0x55665f09ff10 2023-07-14T19:03:56.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.662+0000 7fed597fa700 1 -- 172.21.15.78:0/50777555 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fed6000f810 con 0x55665f09ff10 2023-07-14T19:03:56.676 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed83c44b80 10 client.15264 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:03:56.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed83c44b80 10 client.15264 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:03:56.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed83c44b80 20 client.15264 populate_metadata read hostname 'smithi078' 2023-07-14T19:03:56.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed83c44b80 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x55665f0b4b50 0x55665f0b3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed83c44b80 4 client.15264 mds_command: new command op to 24503 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:03:56.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed83c44b80 1 -- 172.21.15.78:0/50777555 --> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55665f06ed50 con 0x55665f0b4b50 2023-07-14T19:03:56.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed75047700 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x55665f0b4b50 0x55665f0b3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.677 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed597fa700 10 client.15264 ms_handle_connect on v2:172.21.15.88:6838/1379790346 2023-07-14T19:03:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.663+0000 7fed75047700 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x55665f0b4b50 0x55665f0b3f30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=beb45b19eac37452 server_cookie=ddc5af5043de52be in_seq=0 out_seq=0 2023-07-14T19:03:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed597fa700 1 -- 172.21.15.78:0/50777555 <== mds.0 v2:172.21.15.88:6838/1379790346 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55665f06ed50 con 0x55665f0b4b50 2023-07-14T19:03:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed597fa700 10 client.15264 handle_command_reply: tid=0 2023-07-14T19:03:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 client.15264 shutdown 2023-07-14T19:03:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 -- 172.21.15.78:0/50777555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 msgr2=0x55665f0a98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f0a98b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fed60015000 tx=0x7fed60009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 -- 172.21.15.78:0/50777555 shutdown_connections 2023-07-14T19:03:56.678 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x55665f0b4b50 0x55665f0b3f30 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f0b0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed597fa700 0 client.15264 ms_handle_reset on v2:172.21.15.88:6838/1379790346 2023-07-14T19:03:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f0a98b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 --2- 172.21.15.78:0/50777555 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f0a9df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 -- 172.21.15.78:0/50777555 >> 172.21.15.78:0/50777555 conn(0x55665f095630 msgr2=0x55665f099600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 -- 172.21.15.78:0/50777555 shutdown_connections 2023-07-14T19:03:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 1 -- 172.21.15.78:0/50777555 wait complete. 2023-07-14T19:03:56.679 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.664+0000 7fed83c44b80 20 client.15264 trim_cache size 0 max 16384 2023-07-14T19:03:56.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed83c44b80 1 Processor -- start 2023-07-14T19:03:56.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed83c44b80 1 -- start start 2023-07-14T19:03:56.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a4780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed83c44b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665f0a9480 con 0x55665f0a0860 2023-07-14T19:03:56.687 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed83c44b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665ee80de0 con 0x55665f0a1460 2023-07-14T19:03:56.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed83c44b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665ef8cd50 con 0x55665f09ff10 2023-07-14T19:03:56.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75047700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75047700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35834/0 (socket says 172.21.15.78:35834) 2023-07-14T19:03:56.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75047700 1 -- 172.21.15.78:0/4258850565 learned_addr learned my addr 172.21.15.78:0/4258850565 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed76049700 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75848700 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.688 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75047700 1 -- 172.21.15.78:0/4258850565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 msgr2=0x55665f0a4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75047700 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75047700 1 -- 172.21.15.78:0/4258850565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 msgr2=0x55665f0a7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75047700 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75047700 1 -- 172.21.15.78:0/4258850565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55665ee45200 con 0x55665f0a0860 2023-07-14T19:03:56.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed75848700 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:03:56.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.682+0000 7fed76049700 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:56.689 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed75047700 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a4780 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00d620 tx=0x7fed6c00dbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66f465c552baebcb server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed64daa700 1 -- 172.21.15.78:0/4258850565 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c017e60 con 0x55665f0a0860 2023-07-14T19:03:56.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed64daa700 1 -- 172.21.15.78:0/4258850565 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed6c0178b0 con 0x55665f0a0860 2023-07-14T19:03:56.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed64daa700 1 -- 172.21.15.78:0/4258850565 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c0056a0 con 0x55665f0a0860 2023-07-14T19:03:56.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 -- 172.21.15.78:0/4258850565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 msgr2=0x55665f0a4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a4780 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00d620 tx=0x7fed6c00dbe0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 -- 172.21.15.78:0/4258850565 shutdown_connections 2023-07-14T19:03:56.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a1460 0x55665f0a7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.690 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f09ff10 0x55665f0a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 --2- 172.21.15.78:0/4258850565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f0a0860 0x55665f0a4780 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 -- 172.21.15.78:0/4258850565 >> 172.21.15.78:0/4258850565 conn(0x55665f095630 msgr2=0x55665f0b3f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 -- 172.21.15.78:0/4258850565 shutdown_connections 2023-07-14T19:03:56.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.683+0000 7fed83c44b80 1 -- 172.21.15.78:0/4258850565 wait complete. 2023-07-14T19:03:56.691 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 1 -- start start 2023-07-14T19:03:56.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 1 Processor -- start 2023-07-14T19:03:56.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f1087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f10ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f10d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.694 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665f10d7d0 con 0x55665f09ff10 2023-07-14T19:03:56.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665f10d910 con 0x55665f0a0860 2023-07-14T19:03:56.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55665f10da50 con 0x55665f0a1460 2023-07-14T19:03:56.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed83c44b80 10 client.0 _ll_register_callbacks cb 0x55665ef8e710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:03:56.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75848700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f1087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75848700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f1087f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35838/0 (socket says 172.21.15.78:35838) 2023-07-14T19:03:56.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75848700 1 -- 172.21.15.78:0/2748908859 learned_addr learned my addr 172.21.15.78:0/2748908859 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:56.695 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed76049700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f10d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75047700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f10ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75848700 1 -- 172.21.15.78:0/2748908859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 msgr2=0x55665f10d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75848700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f10d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75848700 1 -- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 msgr2=0x55665f10ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:03:56.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75848700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f10ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75848700 1 -- 172.21.15.78:0/2748908859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55665ee1ed90 con 0x55665f09ff10 2023-07-14T19:03:56.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed75047700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:56.696 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.684+0000 7fed76049700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f10d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:03:56.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed75848700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f1087f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fed600098e0 tx=0x7fed600047e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a779824196d6753 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:56.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed597fa700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:03:56.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed597fa700 1 -- 172.21.15.78:0/2748908859 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6000f670 con 0x55665f09ff10 2023-07-14T19:03:56.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed597fa700 1 -- 172.21.15.78:0/2748908859 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fed6000f7d0 con 0x55665f09ff10 2023-07-14T19:03:56.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 1 -- 172.21.15.78:0/2748908859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fed6c00dda0 con 0x55665f09ff10 2023-07-14T19:03:56.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed597fa700 1 -- 172.21.15.78:0/2748908859 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fed6002f490 con 0x55665f09ff10 2023-07-14T19:03:56.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 10 client.15270 fetch_fsmap learned FSMap version 23 2023-07-14T19:03:56.697 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 1 -- 172.21.15.78:0/2748908859 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55665ee1d260 con 0x55665f09ff10 2023-07-14T19:03:56.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed597fa700 1 -- 172.21.15.78:0/2748908859 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed6c00dda0 con 0x55665f09ff10 2023-07-14T19:03:56.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed597fa700 1 -- 172.21.15.78:0/2748908859 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fed60037040 con 0x55665f09ff10 2023-07-14T19:03:56.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 10 client.15270 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:03:56.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 10 client.15270 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:03:56.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 20 client.15270 populate_metadata read hostname 'smithi078' 2023-07-14T19:03:56.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x55665f112400 0x55665f10ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:56.698 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 4 client.15270 mds_command: new command op to 24503 tid=0[{"prefix": "status"}] 2023-07-14T19:03:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.685+0000 7fed83c44b80 1 -- 172.21.15.78:0/2748908859 --> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55665efe9e60 con 0x55665f112400 2023-07-14T19:03:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.686+0000 7fed75047700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x55665f112400 0x55665f10ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.686+0000 7fed75047700 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x55665f112400 0x55665f10ed10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98f3d8dd7e7870f6 server_cookie=5c1011125562cbe1 in_seq=0 out_seq=0 2023-07-14T19:03:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.686+0000 7fed597fa700 10 client.15270 ms_handle_connect on v2:172.21.15.88:6838/1379790346 2023-07-14T19:03:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.686+0000 7fed597fa700 1 -- 172.21.15.78:0/2748908859 <== mds.0 v2:172.21.15.88:6838/1379790346 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55665efe9e60 con 0x55665f112400 2023-07-14T19:03:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.686+0000 7fed597fa700 10 client.15270 handle_command_reply: tid=0 2023-07-14T19:03:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.686+0000 7fed83c44b80 1 client.15270 shutdown 2023-07-14T19:03:56.699 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.686+0000 7fed83c44b80 1 -- 172.21.15.78:0/2748908859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 msgr2=0x55665f1087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.686+0000 7fed83c44b80 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f1087f0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fed600098e0 tx=0x7fed600047e0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 1 -- 172.21.15.78:0/2748908859 shutdown_connections 2023-07-14T19:03:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x55665f112400 0x55665f10ed10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55665f0a1460 0x55665f10d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed597fa700 0 client.15270 ms_handle_reset on v2:172.21.15.88:6838/1379790346 2023-07-14T19:03:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55665f09ff10 0x55665f1087f0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.700 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 1 --2- 172.21.15.78:0/2748908859 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55665f0a0860 0x55665f10ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 1 -- 172.21.15.78:0/2748908859 >> 172.21.15.78:0/2748908859 conn(0x55665f095630 msgr2=0x55665f0a4240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 1 -- 172.21.15.78:0/2748908859 shutdown_connections 2023-07-14T19:03:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 1 -- 172.21.15.78:0/2748908859 wait complete. 2023-07-14T19:03:56.701 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed83c44b80 20 client.15270 trim_cache size 0 max 16384 2023-07-14T19:03:56.701 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:03:56.701 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T19:03:56.702 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-07-14T19:03:56.702 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24503, 2023-07-14T19:03:56.702 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-07-14T19:03:56.702 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-07-14T19:03:56.702 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-07-14T19:03:56.702 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 85.758570114999998, 2023-07-14T19:03:56.702 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2023-07-14T19:03:56.702 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2023-07-14T19:03:56.703 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T19:03:56.703 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 91.770325737999997 2023-07-14T19:03:56.703 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:03:56.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 -- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed5c0a11e0 msgr2=0x7fed5c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed5c0a11e0 0x7fed5c0a3690 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fed6000c010 tx=0x7fed60005fb0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 -- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed70101790 msgr2=0x7fed7019ce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:56.704 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed70101790 0x7fed7019ce30 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fed68007d70 tx=0x7fed6800a340 comp rx=0 tx=0).stop 2023-07-14T19:03:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 -- 172.21.15.78:0/4183247956 shutdown_connections 2023-07-14T19:03:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed5c0a11e0 0x7fed5c0a3690 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed700ffc10 0x7fed70195880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed70101790 0x7fed7019ce30 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 --2- 172.21.15.78:0/4183247956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed70100e00 0x7fed70195dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 -- 172.21.15.78:0/4183247956 >> 172.21.15.78:0/4183247956 conn(0x7fed700fb250 msgr2=0x7fed701042e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:56.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 -- 172.21.15.78:0/4183247956 shutdown_connections 2023-07-14T19:03:56.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:56.687+0000 7fed77aac700 1 -- 172.21.15.78:0/4183247956 wait complete. 2023-07-14T19:03:56.706 INFO:tasks.cephadm:Setup complete, yielding 2023-07-14T19:03:56.707 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T19:03:56.718 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-07-14T19:03:56.718 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- bash -c 'ceph orch status' 2023-07-14T19:03:57.265 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:03:57.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:57 smithi088 ceph-mon[128973]: pgmap v300: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:03:57.482 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:57 smithi078 ceph-mon[127425]: pgmap v300: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:03:57.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:57 smithi102 ceph-mon[129280]: pgmap v300: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2023-07-14T19:03:58.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 -- 172.21.15.78:0/3382946427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88100ef0 msgr2=0x7fed88101340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:58.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 --2- 172.21.15.78:0/3382946427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88100ef0 0x7fed88101340 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fed78009990 tx=0x7fed78009ca0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 -- 172.21.15.78:0/3382946427 shutdown_connections 2023-07-14T19:03:58.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 --2- 172.21.15.78:0/3382946427 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed88101880 0x7fed88108140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 --2- 172.21.15.78:0/3382946427 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed880ffcf0 0x7fed88100100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 --2- 172.21.15.78:0/3382946427 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88100ef0 0x7fed88101340 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 -- 172.21.15.78:0/3382946427 >> 172.21.15.78:0/3382946427 conn(0x7fed880fb360 msgr2=0x7fed880fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:58.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 -- 172.21.15.78:0/3382946427 shutdown_connections 2023-07-14T19:03:58.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.701+0000 7fed8e326700 1 -- 172.21.15.78:0/3382946427 wait complete. 2023-07-14T19:03:58.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8e326700 1 Processor -- start 2023-07-14T19:03:58.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8e326700 1 -- start start 2023-07-14T19:03:58.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8e326700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed880ffcf0 0x7fed881958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:58.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8e326700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed88100ef0 0x7fed88195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8e326700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88101880 0x7fed8819ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8e326700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8819d480 con 0x7fed88101880 2023-07-14T19:03:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8e326700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8819d5c0 con 0x7fed88100ef0 2023-07-14T19:03:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8e326700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8819d750 con 0x7fed880ffcf0 2023-07-14T19:03:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8c8c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88101880 0x7fed8819ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8c8c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88101880 0x7fed8819ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35856/0 (socket says 172.21.15.78:35856) 2023-07-14T19:03:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed8c8c3700 1 -- 172.21.15.78:0/1878205496 learned_addr learned my addr 172.21.15.78:0/1878205496 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:03:58.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.702+0000 7fed877fe700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed88100ef0 0x7fed88195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:58.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed87fff700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed880ffcf0 0x7fed881958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:58.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed8c8c3700 1 -- 172.21.15.78:0/1878205496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed880ffcf0 msgr2=0x7fed881958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:58.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed8c8c3700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed880ffcf0 0x7fed881958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed8c8c3700 1 -- 172.21.15.78:0/1878205496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed88100ef0 msgr2=0x7fed88195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:58.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed8c8c3700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed88100ef0 0x7fed88195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed8c8c3700 1 -- 172.21.15.78:0/1878205496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed780095f0 con 0x7fed88101880 2023-07-14T19:03:58.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed8c8c3700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88101880 0x7fed8819ceb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fed8000ef60 tx=0x7fed8000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed857fa700 1 -- 172.21.15.78:0/1878205496 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed80018070 con 0x7fed88101880 2023-07-14T19:03:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed857fa700 1 -- 172.21.15.78:0/1878205496 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed8000f040 con 0x7fed88101880 2023-07-14T19:03:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8819d9d0 con 0x7fed88101880 2023-07-14T19:03:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed857fa700 1 -- 172.21.15.78:0/1878205496 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed8000cc60 con 0x7fed88101880 2023-07-14T19:03:58.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.703+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8819de80 con 0x7fed88101880 2023-07-14T19:03:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.704+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8810b360 con 0x7fed88101880 2023-07-14T19:03:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.705+0000 7fed857fa700 1 -- 172.21.15.78:0/1878205496 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fed80004ad0 con 0x7fed88101880 2023-07-14T19:03:58.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.706+0000 7fed857fa700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed700a11e0 0x7fed700a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:03:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.706+0000 7fed87fff700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed700a11e0 0x7fed700a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:03:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.706+0000 7fed857fa700 1 -- 172.21.15.78:0/1878205496 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fed800cc8b0 con 0x7fed88101880 2023-07-14T19:03:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.706+0000 7fed87fff700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed700a11e0 0x7fed700a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fed88100d50 tx=0x7fed7c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:03:58.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.710+0000 7fed857fa700 1 -- 172.21.15.78:0/1878205496 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fed80095bc0 con 0x7fed88101880 2023-07-14T19:03:58.918 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.914+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fed88060a80 con 0x7fed700a11e0 2023-07-14T19:03:58.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.916+0000 7fed857fa700 1 -- 172.21.15.78:0/1878205496 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fed88060a80 con 0x7fed700a11e0 2023-07-14T19:03:58.919 INFO:teuthology.orchestra.run.smithi078.stdout:Backend: cephadm 2023-07-14T19:03:58.919 INFO:teuthology.orchestra.run.smithi078.stdout:Available: Yes 2023-07-14T19:03:58.919 INFO:teuthology.orchestra.run.smithi078.stdout:Paused: No 2023-07-14T19:03:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed700a11e0 msgr2=0x7fed700a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed700a11e0 0x7fed700a3690 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fed88100d50 tx=0x7fed7c009260 comp rx=0 tx=0).stop 2023-07-14T19:03:58.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88101880 msgr2=0x7fed8819ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:03:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88101880 0x7fed8819ceb0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fed8000ef60 tx=0x7fed8000ef90 comp rx=0 tx=0).stop 2023-07-14T19:03:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 shutdown_connections 2023-07-14T19:03:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fed700a11e0 0x7fed700a3690 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed88101880 0x7fed8819ceb0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed880ffcf0 0x7fed881958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 --2- 172.21.15.78:0/1878205496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed88100ef0 0x7fed88195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:03:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 >> 172.21.15.78:0/1878205496 conn(0x7fed880fb360 msgr2=0x7fed880fd620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:03:58.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 shutdown_connections 2023-07-14T19:03:58.923 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:03:58.918+0000 7fed8e326700 1 -- 172.21.15.78:0/1878205496 wait complete. 2023-07-14T19:03:59.315 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ps' 2023-07-14T19:03:59.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:03:59 smithi078 ceph-mon[127425]: pgmap v301: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:59.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:03:59 smithi102 ceph-mon[129280]: pgmap v301: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:59.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:03:59 smithi088 ceph-mon[128973]: pgmap v301: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2023-07-14T19:03:59.867 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:04:00.484 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:00 smithi078 ceph-mon[127425]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:00.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:00 smithi102 ceph-mon[129280]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:00.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:00 smithi088 ceph-mon[128973]: from='client.15276 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:01.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 -- 172.21.15.78:0/4129562383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641007a0 msgr2=0x7f7764102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:01.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 --2- 172.21.15.78:0/4129562383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641007a0 0x7f7764102b80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f7758009990 tx=0x7f7758009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 -- 172.21.15.78:0/4129562383 shutdown_connections 2023-07-14T19:04:01.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 --2- 172.21.15.78:0/4129562383 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77641059e0 0x7f7764107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 --2- 172.21.15.78:0/4129562383 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641007a0 0x7f7764102b80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 --2- 172.21.15.78:0/4129562383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77641030c0 0x7f77641054a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.267 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 -- 172.21.15.78:0/4129562383 >> 172.21.15.78:0/4129562383 conn(0x7f77640fa1e0 msgr2=0x7f77640fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:01.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 -- 172.21.15.78:0/4129562383 shutdown_connections 2023-07-14T19:04:01.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.209+0000 7f776bcd2700 1 -- 172.21.15.78:0/4129562383 wait complete. 2023-07-14T19:04:01.268 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776bcd2700 1 Processor -- start 2023-07-14T19:04:01.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776bcd2700 1 -- start start 2023-07-14T19:04:01.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776bcd2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77641007a0 0x7f77641958e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:01.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776bcd2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77641030c0 0x7f7764195e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:01.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776bcd2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641059e0 0x7f776419cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:01.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776bcd2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776419d470 con 0x7f77641059e0 2023-07-14T19:04:01.269 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776bcd2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776419d5b0 con 0x7f77641007a0 2023-07-14T19:04:01.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776bcd2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776419d740 con 0x7f77641030c0 2023-07-14T19:04:01.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776a26f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641059e0 0x7f776419cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:01.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776a26f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641059e0 0x7f776419cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35868/0 (socket says 172.21.15.78:35868) 2023-07-14T19:04:01.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.210+0000 7f776a26f700 1 -- 172.21.15.78:0/1855297755 learned_addr learned my addr 172.21.15.78:0/1855297755 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:01.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f7769a6e700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77641007a0 0x7f77641958e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:01.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776926d700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77641030c0 0x7f7764195e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:01.270 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776a26f700 1 -- 172.21.15.78:0/1855297755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77641030c0 msgr2=0x7f7764195e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:01.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776a26f700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77641030c0 0x7f7764195e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776a26f700 1 -- 172.21.15.78:0/1855297755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77641007a0 msgr2=0x7f77641958e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:01.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776a26f700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77641007a0 0x7f77641958e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776a26f700 1 -- 172.21.15.78:0/1855297755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77580095f0 con 0x7f77641059e0 2023-07-14T19:04:01.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776a26f700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641059e0 0x7f776419cea0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f775c007ee0 tx=0x7f775c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:01.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f7756ffd700 1 -- 172.21.15.78:0/1855297755 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c004db0 con 0x7f77641059e0 2023-07-14T19:04:01.271 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f7756ffd700 1 -- 172.21.15.78:0/1855297755 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f775c00f040 con 0x7f77641059e0 2023-07-14T19:04:01.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f7756ffd700 1 -- 172.21.15.78:0/1855297755 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f775c005870 con 0x7f77641059e0 2023-07-14T19:04:01.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776419da20 con 0x7f77641059e0 2023-07-14T19:04:01.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.211+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776419ded0 con 0x7f77641059e0 2023-07-14T19:04:01.272 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.212+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77640fbd50 con 0x7f77641059e0 2023-07-14T19:04:01.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.213+0000 7f7756ffd700 1 -- 172.21.15.78:0/1855297755 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 17) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f775c005000 con 0x7f77641059e0 2023-07-14T19:04:01.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.214+0000 7f7756ffd700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77500a11e0 0x7f77500a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:01.274 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.214+0000 7f7769a6e700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77500a11e0 0x7f77500a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:01.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.214+0000 7f7756ffd700 1 -- 172.21.15.78:0/1855297755 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f775c0ccfa0 con 0x7f77641059e0 2023-07-14T19:04:01.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.214+0000 7f7769a6e700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77500a11e0 0x7f77500a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f775800b5c0 tx=0x7f7758005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:01.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.217+0000 7f7756ffd700 1 -- 172.21.15.78:0/1855297755 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f775c096280 con 0x7f77641059e0 2023-07-14T19:04:01.275 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.230+0000 7f7756ffd700 1 -- 172.21.15.78:0/1855297755 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f775c096650 con 0x7f77641059e0 2023-07-14T19:04:01.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:01 smithi078 ceph-mon[127425]: pgmap v302: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:04:01.420 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.417+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f7764060a80 con 0x7f77500a11e0 2023-07-14T19:04:01.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.428+0000 7f7756ffd700 1 -- 172.21.15.78:0/1855297755 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4185 (secure 0 0 0) 0x7f7764060a80 con 0x7f77500a11e0 2023-07-14T19:04:01.433 INFO:teuthology.orchestra.run.smithi078.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-07-14T19:04:01.433 INFO:teuthology.orchestra.run.smithi078.stdout:mds.a smithi078 running (99s) - 98s 17.3M - 18.0.0-4940-g81c9387d 219b180907a8 196f8f1f3ef2 2023-07-14T19:04:01.433 INFO:teuthology.orchestra.run.smithi078.stdout:mds.b smithi088 running (96s) - 96s 29.9M - 18.0.0-4940-g81c9387d 219b180907a8 b4f6c42092d1 2023-07-14T19:04:01.433 INFO:teuthology.orchestra.run.smithi078.stdout:mds.c smithi102 running (107s) - 107s 20.1M - 18.0.0-4940-g81c9387d 219b180907a8 aecba148408b 2023-07-14T19:04:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:mds.d smithi078 running (114s) - 113s 14.0M - 18.0.0-4940-g81c9387d 219b180907a8 aa7b4c49d791 2023-07-14T19:04:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:mds.e smithi088 running (112s) - 111s 17.6M - 18.0.0-4940-g81c9387d 219b180907a8 7ea1cb288dd6 2023-07-14T19:04:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:mds.f smithi102 running (109s) - 109s 19.0M - 18.0.0-4940-g81c9387d 219b180907a8 383a23e43256 2023-07-14T19:04:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:mds.g smithi078 running (101s) - 100s 16.4M - 18.0.0-4940-g81c9387d 219b180907a8 8a0c7d65f21b 2023-07-14T19:04:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:mds.h smithi088 running (105s) - 105s 18.6M - 18.0.0-4940-g81c9387d 219b180907a8 a7cfc0450c09 2023-07-14T19:04:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:mds.i smithi102 running (103s) - 103s 18.5M - 18.0.0-4940-g81c9387d 219b180907a8 6e63d978d0ab 2023-07-14T19:04:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:mds.j smithi078 running (116s) - 116s 14.8M - 18.0.0-4940-g81c9387d 219b180907a8 29eaebd80c3d 2023-07-14T19:04:01.434 INFO:teuthology.orchestra.run.smithi078.stdout:mds.k smithi088 running (118s) - 118s 17.2M - 18.0.0-4940-g81c9387d 219b180907a8 c700cf98ed5f 2023-07-14T19:04:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:mds.l smithi102 running (2m) - 2m 17.3M - 18.0.0-4940-g81c9387d 219b180907a8 eaa63461d941 2023-07-14T19:04:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.x smithi078 *:9283,8765 running (8m) - 8m 517M - 18.0.0-4940-g81c9387d 219b180907a8 0b3018b2a03c 2023-07-14T19:04:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.y smithi088 *:8443,8765 running (6m) - 6m 450M - 18.0.0-4940-g81c9387d 219b180907a8 0363a37ae0f0 2023-07-14T19:04:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:mgr.z smithi102 *:8443,8765 running (6m) - 6m 448M - 18.0.0-4940-g81c9387d 219b180907a8 639857f58158 2023-07-14T19:04:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:mon.a smithi078 running (8m) - 8m 91.7M 2048M 18.0.0-4940-g81c9387d 219b180907a8 858abc030ff9 2023-07-14T19:04:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:mon.b smithi088 running (6m) - 6m 59.7M 2048M 18.0.0-4940-g81c9387d 219b180907a8 b92907992a5a 2023-07-14T19:04:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:mon.c smithi102 running (6m) - 6m 61.0M 2048M 18.0.0-4940-g81c9387d 219b180907a8 2374cf87c460 2023-07-14T19:04:01.435 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 smithi078 running (6m) - 6m 70.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 dcd8654a964e 2023-07-14T19:04:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 smithi088 running (5m) - 5m 79.7M 4292M 18.0.0-4940-g81c9387d 219b180907a8 a51523a22746 2023-07-14T19:04:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 smithi102 running (5m) - 5m 75.1M 4292M 18.0.0-4940-g81c9387d 219b180907a8 66a41164110f 2023-07-14T19:04:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 smithi078 running (5m) - 5m 65.8M 4292M 18.0.0-4940-g81c9387d 219b180907a8 99cf378fb968 2023-07-14T19:04:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 smithi088 running (4m) - 4m 65.9M 4292M 18.0.0-4940-g81c9387d 219b180907a8 f493b100488c 2023-07-14T19:04:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 smithi102 running (4m) - 4m 68.5M 4292M 18.0.0-4940-g81c9387d 219b180907a8 ab0524b762b3 2023-07-14T19:04:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 smithi078 running (4m) - 4m 61.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 52afbccadfcb 2023-07-14T19:04:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 smithi088 running (3m) - 3m 63.8M 4292M 18.0.0-4940-g81c9387d 219b180907a8 5b5014146286 2023-07-14T19:04:01.436 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 smithi102 running (3m) - 3m 62.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 5abd1af2ad6a 2023-07-14T19:04:01.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 smithi078 running (3m) - 3m 60.2M 4292M 18.0.0-4940-g81c9387d 219b180907a8 e3a31e365fbd 2023-07-14T19:04:01.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 smithi088 running (2m) - 2m 62.9M 4292M 18.0.0-4940-g81c9387d 219b180907a8 8348f8105b0e 2023-07-14T19:04:01.437 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 smithi102 running (2m) - 2m 57.1M 4292M 18.0.0-4940-g81c9387d 219b180907a8 f4f24ca27ca3 2023-07-14T19:04:01.437 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.430+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77500a11e0 msgr2=0x7f77500a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:01.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.430+0000 7f776bcd2700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77500a11e0 0x7f77500a3690 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f775800b5c0 tx=0x7f7758005fb0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641059e0 msgr2=0x7f776419cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:01.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641059e0 0x7f776419cea0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f775c007ee0 tx=0x7f775c00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 shutdown_connections 2023-07-14T19:04:01.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77500a11e0 0x7f77500a3690 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.438 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77641007a0 0x7f77641958e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77641059e0 0x7f776419cea0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 --2- 172.21.15.78:0/1855297755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77641030c0 0x7f7764195e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:01.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 >> 172.21.15.78:0/1855297755 conn(0x7f77640fa1e0 msgr2=0x7f7764101400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:01.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 shutdown_connections 2023-07-14T19:04:01.439 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:01.431+0000 7f776bcd2700 1 -- 172.21.15.78:0/1855297755 wait complete. 2023-07-14T19:04:01.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:01 smithi102 ceph-mon[129280]: pgmap v302: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:04:01.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:01 smithi088 ceph-mon[128973]: pgmap v302: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:04:01.928 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- bash -c 'ceph orch ls' 2023-07-14T19:04:02.480 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:04:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:02 smithi078 ceph-mon[127425]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T19:04:02.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:02 smithi078 ceph-mon[127425]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:02.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:02 smithi102 ceph-mon[129280]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T19:04:02.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:02 smithi102 ceph-mon[129280]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:02.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:02 smithi088 ceph-mon[128973]: mgrmap e18: x(active, since 7m), standbys: z, y 2023-07-14T19:04:02.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:02 smithi088 ceph-mon[128973]: from='client.15282 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:03.243 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:03 smithi078 ceph-mon[127425]: pgmap v303: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:04:03.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:03 smithi102 ceph-mon[129280]: pgmap v303: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:04:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.650+0000 7f06f39fc700 1 -- 172.21.15.78:0/3098789562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec1001c0 msgr2=0x7f06ec1005d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:03.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.650+0000 7f06f39fc700 1 --2- 172.21.15.78:0/3098789562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec1001c0 0x7f06ec1005d0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f06dc009990 tx=0x7f06dc009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.650+0000 7f06f39fc700 1 -- 172.21.15.78:0/3098789562 shutdown_connections 2023-07-14T19:04:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.650+0000 7f06f39fc700 1 --2- 172.21.15.78:0/3098789562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ec0ff870 0x7f06ec107120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.650+0000 7f06f39fc700 1 --2- 172.21.15.78:0/3098789562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec1001c0 0x7f06ec1005d0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.650+0000 7f06f39fc700 1 --2- 172.21.15.78:0/3098789562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ec0feec0 0x7f06ec0ff330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.650+0000 7f06f39fc700 1 -- 172.21.15.78:0/3098789562 >> 172.21.15.78:0/3098789562 conn(0x7f06ec0f99c0 msgr2=0x7f06ec0fbdd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.650+0000 7f06f39fc700 1 -- 172.21.15.78:0/3098789562 shutdown_connections 2023-07-14T19:04:03.655 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.651+0000 7f06f39fc700 1 -- 172.21.15.78:0/3098789562 wait complete. 2023-07-14T19:04:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.651+0000 7f06f39fc700 1 Processor -- start 2023-07-14T19:04:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.651+0000 7f06f39fc700 1 -- start start 2023-07-14T19:04:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.651+0000 7f06f39fc700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ec0feec0 0x7f06ec1948a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:03.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f39fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec0ff870 0x7f06ec194de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f39fc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ec1001c0 0x7f06ec19be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f39fc700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec19c430 con 0x7f06ec0ff870 2023-07-14T19:04:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f39fc700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec19c570 con 0x7f06ec0feec0 2023-07-14T19:04:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f39fc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06ec19c700 con 0x7f06ec1001c0 2023-07-14T19:04:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec0ff870 0x7f06ec194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec0ff870 0x7f06ec194de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35880/0 (socket says 172.21.15.78:35880) 2023-07-14T19:04:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 -- 172.21.15.78:0/2309696102 learned_addr learned my addr 172.21.15.78:0/2309696102 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:03.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f1798700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ec0feec0 0x7f06ec1948a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f1f99700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ec1001c0 0x7f06ec19be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 -- 172.21.15.78:0/2309696102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ec1001c0 msgr2=0x7f06ec19be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ec1001c0 0x7f06ec19be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 -- 172.21.15.78:0/2309696102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ec0feec0 msgr2=0x7f06ec1948a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ec0feec0 0x7f06ec1948a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 -- 172.21.15.78:0/2309696102 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06dc0095f0 con 0x7f06ec0ff870 2023-07-14T19:04:03.659 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f1f99700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ec1001c0 0x7f06ec19be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f0f97700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec0ff870 0x7f06ec194de0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f06e800b560 tx=0x7f06e800b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06e27fc700 1 -- 172.21.15.78:0/2309696102 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8011830 con 0x7f06ec0ff870 2023-07-14T19:04:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06e27fc700 1 -- 172.21.15.78:0/2309696102 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f06e8011e20 con 0x7f06ec0ff870 2023-07-14T19:04:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06ec19c9e0 con 0x7f06ec0ff870 2023-07-14T19:04:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06e27fc700 1 -- 172.21.15.78:0/2309696102 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e8010dd0 con 0x7f06ec0ff870 2023-07-14T19:04:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.652+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06ec19cfb0 con 0x7f06ec0ff870 2023-07-14T19:04:03.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.653+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06d0005320 con 0x7f06ec0ff870 2023-07-14T19:04:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.654+0000 7f06e27fc700 1 -- 172.21.15.78:0/2309696102 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f06e8010380 con 0x7f06ec0ff870 2023-07-14T19:04:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.655+0000 7f06e27fc700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f06d80a11d0 0x7f06d80a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:03.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.655+0000 7f06f1798700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f06d80a11d0 0x7f06d80a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.655+0000 7f06e27fc700 1 -- 172.21.15.78:0/2309696102 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f06e80cc5b0 con 0x7f06ec0ff870 2023-07-14T19:04:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.655+0000 7f06f1798700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f06d80a11d0 0x7f06d80a3680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00b5c0 tx=0x7f06dc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:03.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.659+0000 7f06e27fc700 1 -- 172.21.15.78:0/2309696102 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f06e80959f0 con 0x7f06ec0ff870 2023-07-14T19:04:03.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:03 smithi088 ceph-mon[128973]: pgmap v303: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 6 op/s 2023-07-14T19:04:03.857 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.853+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f06d0000bf0 con 0x7f06d80a11d0 2023-07-14T19:04:03.861 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.858+0000 7f06e27fc700 1 -- 172.21.15.78:0/2309696102 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f06d0000bf0 con 0x7f06d80a11d0 2023-07-14T19:04:03.862 INFO:teuthology.orchestra.run.smithi078.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-07-14T19:04:03.862 INFO:teuthology.orchestra.run.smithi078.stdout:mds.all 12/12 - 2m smithi078=a;smithi078=d;smithi078=g;smithi078=j;smithi088=b;smithi088=e;smithi088=h;smithi088=k;smithi102=c;smithi102=f;smithi102=i;smithi102=l;count:12 2023-07-14T19:04:03.862 INFO:teuthology.orchestra.run.smithi078.stdout:mgr 3/3 - 6m smithi078=x;smithi088=y;smithi102=z;count:3 2023-07-14T19:04:03.862 INFO:teuthology.orchestra.run.smithi078.stdout:mon 3/3 - 6m smithi078:172.21.15.78=a;smithi088:172.21.15.88=b;smithi102:172.21.15.102=c;count:3 2023-07-14T19:04:03.862 INFO:teuthology.orchestra.run.smithi078.stdout:osd 12 - - 2023-07-14T19:04:03.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f06d80a11d0 msgr2=0x7f06d80a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:03.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f06d80a11d0 0x7f06d80a3680 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f06dc00b5c0 tx=0x7f06dc005fb0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec0ff870 msgr2=0x7f06ec194de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:03.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec0ff870 0x7f06ec194de0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f06e800b560 tx=0x7f06e800b870 comp rx=0 tx=0).stop 2023-07-14T19:04:03.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 shutdown_connections 2023-07-14T19:04:03.864 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f06d80a11d0 0x7f06d80a3680 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f06ec0feec0 0x7f06ec1948a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f06ec1001c0 0x7f06ec19be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 --2- 172.21.15.78:0/2309696102 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f06ec0ff870 0x7f06ec194de0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 >> 172.21.15.78:0/2309696102 conn(0x7f06ec0f99c0 msgr2=0x7f06ec0fbc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 shutdown_connections 2023-07-14T19:04:03.865 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:03.860+0000 7f06f39fc700 1 -- 172.21.15.78:0/2309696102 wait complete. 2023-07-14T19:04:04.330 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- bash -c 'ceph orch host ls' 2023-07-14T19:04:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:04:05.408 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:05 smithi078 ceph-mon[127425]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:05.409 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:05 smithi078 ceph-mon[127425]: pgmap v304: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-07-14T19:04:05.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:05 smithi102 ceph-mon[129280]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:05.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:05 smithi102 ceph-mon[129280]: pgmap v304: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-07-14T19:04:05.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:05 smithi088 ceph-mon[128973]: from='client.15288 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:05.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:05 smithi088 ceph-mon[128973]: pgmap v304: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 511 B/s wr, 6 op/s 2023-07-14T19:04:06.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 -- 172.21.15.78:0/3188427399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e40ffcc0 msgr2=0x7fc3e41000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:06.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 --2- 172.21.15.78:0/3188427399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e40ffcc0 0x7fc3e41000d0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8009990 tx=0x7fc3d8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 -- 172.21.15.78:0/3188427399 shutdown_connections 2023-07-14T19:04:06.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 --2- 172.21.15.78:0/3188427399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e4101850 0x7fc3e4108110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 --2- 172.21.15.78:0/3188427399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e40ffcc0 0x7fc3e41000d0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 --2- 172.21.15.78:0/3188427399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e4100ec0 0x7fc3e4101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 -- 172.21.15.78:0/3188427399 >> 172.21.15.78:0/3188427399 conn(0x7fc3e40fb2d0 msgr2=0x7fc3e40fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:06.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 -- 172.21.15.78:0/3188427399 shutdown_connections 2023-07-14T19:04:06.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.217+0000 7fc3e9075700 1 -- 172.21.15.78:0/3188427399 wait complete. 2023-07-14T19:04:06.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.218+0000 7fc3e9075700 1 Processor -- start 2023-07-14T19:04:06.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.218+0000 7fc3e9075700 1 -- start start 2023-07-14T19:04:06.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.218+0000 7fc3e9075700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e40ffcc0 0x7fc3e40743e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:06.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e9075700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e4100ec0 0x7fc3e4072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:06.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e9075700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e4101850 0x7fc3e4072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:06.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e9075700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e40736f0 con 0x7fc3e4101850 2023-07-14T19:04:06.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e9075700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e4073860 con 0x7fc3e4100ec0 2023-07-14T19:04:06.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e9075700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3e419f980 con 0x7fc3e40ffcc0 2023-07-14T19:04:06.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e4101850 0x7fc3e4072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:06.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e4101850 0x7fc3e4072f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35896/0 (socket says 172.21.15.78:35896) 2023-07-14T19:04:06.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e2d9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e40ffcc0 0x7fc3e40743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:06.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 -- 172.21.15.78:0/1729171498 learned_addr learned my addr 172.21.15.78:0/1729171498 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:06.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e259c700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e4100ec0 0x7fc3e4072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:06.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 -- 172.21.15.78:0/1729171498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e40ffcc0 msgr2=0x7fc3e40743e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:06.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e40ffcc0 0x7fc3e40743e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 -- 172.21.15.78:0/1729171498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e4100ec0 msgr2=0x7fc3e4072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e4100ec0 0x7fc3e4072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 -- 172.21.15.78:0/1729171498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d80095f0 con 0x7fc3e4101850 2023-07-14T19:04:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e259c700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e4100ec0 0x7fc3e4072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e359e700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e4101850 0x7fc3e4072f70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc007ee0 tx=0x7fc3dc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3cbfff700 1 -- 172.21.15.78:0/1729171498 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00ddd0 con 0x7fc3e4101850 2023-07-14T19:04:06.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3e419fc60 con 0x7fc3e4101850 2023-07-14T19:04:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3cbfff700 1 -- 172.21.15.78:0/1729171498 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc3dc00f040 con 0x7fc3e4101850 2023-07-14T19:04:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3cbfff700 1 -- 172.21.15.78:0/1729171498 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0047d0 con 0x7fc3e4101850 2023-07-14T19:04:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.219+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3e41a0160 con 0x7fc3e4101850 2023-07-14T19:04:06.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.220+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3e410b330 con 0x7fc3e4101850 2023-07-14T19:04:06.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.221+0000 7fc3cbfff700 1 -- 172.21.15.78:0/1729171498 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc3dc003a00 con 0x7fc3e4101850 2023-07-14T19:04:06.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.222+0000 7fc3cbfff700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3cc0a1150 0x7fc3cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:06.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.222+0000 7fc3e2d9d700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3cc0a1150 0x7fc3cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:06.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.222+0000 7fc3cbfff700 1 -- 172.21.15.78:0/1729171498 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc3dc0cc4d0 con 0x7fc3e4101850 2023-07-14T19:04:06.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.223+0000 7fc3e2d9d700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3cc0a1150 0x7fc3cc0a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800b5c0 tx=0x7fc3d8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:06.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.226+0000 7fc3cbfff700 1 -- 172.21.15.78:0/1729171498 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc3dc04e780 con 0x7fc3e4101850 2023-07-14T19:04:06.426 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.423+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3e4060a80 con 0x7fc3cc0a1150 2023-07-14T19:04:06.428 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.425+0000 7fc3cbfff700 1 -- 172.21.15.78:0/1729171498 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7fc3e4060a80 con 0x7fc3cc0a1150 2023-07-14T19:04:06.429 INFO:teuthology.orchestra.run.smithi078.stdout:HOST ADDR LABELS STATUS 2023-07-14T19:04:06.429 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 172.21.15.78 2023-07-14T19:04:06.429 INFO:teuthology.orchestra.run.smithi078.stdout:smithi088 172.21.15.88 2023-07-14T19:04:06.429 INFO:teuthology.orchestra.run.smithi078.stdout:smithi102 172.21.15.102 2023-07-14T19:04:06.429 INFO:teuthology.orchestra.run.smithi078.stdout:3 hosts in cluster 2023-07-14T19:04:06.430 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3cc0a1150 msgr2=0x7fc3cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:06.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3cc0a1150 0x7fc3cc0a3600 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800b5c0 tx=0x7fc3d8005fb0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e4101850 msgr2=0x7fc3e4072f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:06.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e4101850 0x7fc3e4072f70 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc007ee0 tx=0x7fc3dc00d3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.431 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 shutdown_connections 2023-07-14T19:04:06.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3cc0a1150 0x7fc3cc0a3600 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3e4101850 0x7fc3e4072f70 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3e40ffcc0 0x7fc3e40743e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 --2- 172.21.15.78:0/1729171498 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3e4100ec0 0x7fc3e4072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:06.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 >> 172.21.15.78:0/1729171498 conn(0x7fc3e40fb2d0 msgr2=0x7fc3e40fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:06.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 shutdown_connections 2023-07-14T19:04:06.432 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:06.427+0000 7fc3e9075700 1 -- 172.21.15.78:0/1729171498 wait complete. 2023-07-14T19:04:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:06 smithi078 ceph-mon[127425]: pgmap v305: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:06.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:06 smithi078 ceph-mon[127425]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:06.833 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- bash -c 'ceph orch device ls' 2023-07-14T19:04:06.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:06 smithi088 ceph-mon[128973]: pgmap v305: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:06.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:06 smithi088 ceph-mon[128973]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:07.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:06 smithi102 ceph-mon[129280]: pgmap v305: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:07.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:06 smithi102 ceph-mon[129280]: from='client.15294 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:07.383 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:04:08.836 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 -- 172.21.15.78:0/2902083447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0ffdb0 msgr2=0x7fe22c100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:08.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 --2- 172.21.15.78:0/2902083447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0ffdb0 0x7fe22c100200 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fe228009990 tx=0x7fe228009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:08.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 -- 172.21.15.78:0/2902083447 shutdown_connections 2023-07-14T19:04:08.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 --2- 172.21.15.78:0/2902083447 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe22c0febb0 0x7fe22c0fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:08.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 --2- 172.21.15.78:0/2902083447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe22c100740 0x7fe22c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:08.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 --2- 172.21.15.78:0/2902083447 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0ffdb0 0x7fe22c100200 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:08.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 -- 172.21.15.78:0/2902083447 >> 172.21.15.78:0/2902083447 conn(0x7fe22c0fa1e0 msgr2=0x7fe22c0fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:08.837 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 -- 172.21.15.78:0/2902083447 shutdown_connections 2023-07-14T19:04:08.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.831+0000 7fe233f0b700 1 -- 172.21.15.78:0/2902083447 wait complete. 2023-07-14T19:04:08.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe233f0b700 1 Processor -- start 2023-07-14T19:04:08.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe233f0b700 1 -- start start 2023-07-14T19:04:08.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe233f0b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0febb0 0x7fe22c068c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:08.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe233f0b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe22c0ffdb0 0x7fe22c19a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:08.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe233f0b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe22c100740 0x7fe22c19cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:08.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe233f0b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c19d430 con 0x7fe22c0febb0 2023-07-14T19:04:08.838 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe233f0b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c19d5a0 con 0x7fe22c100740 2023-07-14T19:04:08.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe233f0b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe22c19d760 con 0x7fe22c0ffdb0 2023-07-14T19:04:08.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe231ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0febb0 0x7fe22c068c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:08.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe231ca7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0febb0 0x7fe22c068c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35904/0 (socket says 172.21.15.78:35904) 2023-07-14T19:04:08.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe231ca7700 1 -- 172.21.15.78:0/428366509 learned_addr learned my addr 172.21.15.78:0/428366509 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:08.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe2314a6700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe22c0ffdb0 0x7fe22c19a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:08.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe2324a8700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe22c100740 0x7fe22c19cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:08.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe231ca7700 1 -- 172.21.15.78:0/428366509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe22c0ffdb0 msgr2=0x7fe22c19a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:08.839 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe231ca7700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe22c0ffdb0 0x7fe22c19a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:08.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe231ca7700 1 -- 172.21.15.78:0/428366509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe22c100740 msgr2=0x7fe22c19cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:08.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe231ca7700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe22c100740 0x7fe22c19cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:08.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe231ca7700 1 -- 172.21.15.78:0/428366509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2280095f0 con 0x7fe22c0febb0 2023-07-14T19:04:08.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.832+0000 7fe2324a8700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe22c100740 0x7fe22c19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:08.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.833+0000 7fe231ca7700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0febb0 0x7fe22c068c80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe22000c970 tx=0x7fe22000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:08.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.833+0000 7fe21effd700 1 -- 172.21.15.78:0/428366509 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220004220 con 0x7fe22c0febb0 2023-07-14T19:04:08.840 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.833+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe22c19da40 con 0x7fe22c0febb0 2023-07-14T19:04:08.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.833+0000 7fe21effd700 1 -- 172.21.15.78:0/428366509 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe220007d30 con 0x7fe22c0febb0 2023-07-14T19:04:08.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.833+0000 7fe21effd700 1 -- 172.21.15.78:0/428366509 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe220003b30 con 0x7fe22c0febb0 2023-07-14T19:04:08.841 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.833+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe22c19df90 con 0x7fe22c0febb0 2023-07-14T19:04:08.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.834+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe22c105b40 con 0x7fe22c0febb0 2023-07-14T19:04:08.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.835+0000 7fe21effd700 1 -- 172.21.15.78:0/428366509 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe220003d50 con 0x7fe22c0febb0 2023-07-14T19:04:08.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.835+0000 7fe21effd700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe2180a1150 0x7fe2180a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:08.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.836+0000 7fe2314a6700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe2180a1150 0x7fe2180a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:08.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.836+0000 7fe21effd700 1 -- 172.21.15.78:0/428366509 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe2200ccc10 con 0x7fe22c0febb0 2023-07-14T19:04:08.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.836+0000 7fe2314a6700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe2180a1150 0x7fe2180a3600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe22c0691e0 tx=0x7fe228005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:08.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:08.839+0000 7fe21effd700 1 -- 172.21.15.78:0/428366509 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe220096050 con 0x7fe22c0febb0 2023-07-14T19:04:09.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.032+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe22c060a80 con 0x7fe2180a1150 2023-07-14T19:04:09.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.034+0000 7fe21effd700 1 -- 172.21.15.78:0/428366509 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+492 (secure 0 0 0) 0x7fe22c060a80 con 0x7fe2180a1150 2023-07-14T19:04:09.038 INFO:teuthology.orchestra.run.smithi078.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-07-14T19:04:09.038 INFO:teuthology.orchestra.run.smithi078.stdout:smithi078 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400QR400BGN 372G No 3m ago LVM detected, locked 2023-07-14T19:04:09.038 INFO:teuthology.orchestra.run.smithi078.stdout:smithi088 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400XD400BGN 372G No 2m ago LVM detected, locked 2023-07-14T19:04:09.038 INFO:teuthology.orchestra.run.smithi078.stdout:smithi102 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VK400BGN 372G No 2m ago LVM detected, locked 2023-07-14T19:04:09.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe2180a1150 msgr2=0x7fe2180a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:09.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe2180a1150 0x7fe2180a3600 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe22c0691e0 tx=0x7fe228005fb0 comp rx=0 tx=0).stop 2023-07-14T19:04:09.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0febb0 msgr2=0x7fe22c068c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:09.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0febb0 0x7fe22c068c80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fe22000c970 tx=0x7fe22000cc80 comp rx=0 tx=0).stop 2023-07-14T19:04:09.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 shutdown_connections 2023-07-14T19:04:09.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe2180a1150 0x7fe2180a3600 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:09.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe22c100740 0x7fe22c19cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:09.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe22c0febb0 0x7fe22c068c80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:09.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 --2- 172.21.15.78:0/428366509 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe22c0ffdb0 0x7fe22c19a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:09.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 >> 172.21.15.78:0/428366509 conn(0x7fe22c0fa1e0 msgr2=0x7fe22c0fc490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:09.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 shutdown_connections 2023-07-14T19:04:09.041 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:09.036+0000 7fe233f0b700 1 -- 172.21.15.78:0/428366509 wait complete. 2023-07-14T19:04:09.443 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:09 smithi078 ceph-mon[127425]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:09.444 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-07-14T19:04:09.453 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-07-14T19:04:09.453 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- bash -c 'ceph fs dump' 2023-07-14T19:04:09.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:09 smithi088 ceph-mon[128973]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:09.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:09 smithi102 ceph-mon[129280]: pgmap v306: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:10.009 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:04:10.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:10 smithi088 ceph-mon[128973]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:10.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:10 smithi078 ceph-mon[127425]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:10.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:10 smithi102 ceph-mon[129280]: from='client.15300 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:11.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 -- 172.21.15.78:0/2975637562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 msgr2=0x7fa600100200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:11.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 --2- 172.21.15.78:0/2975637562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 0x7fa600100200 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009990 tx=0x7fa5f0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 -- 172.21.15.78:0/2975637562 shutdown_connections 2023-07-14T19:04:11.352 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 --2- 172.21.15.78:0/2975637562 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa6000febb0 0x7fa6000fefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 --2- 172.21.15.78:0/2975637562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa600100740 0x7fa600107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 --2- 172.21.15.78:0/2975637562 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 0x7fa600100200 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 -- 172.21.15.78:0/2975637562 >> 172.21.15.78:0/2975637562 conn(0x7fa6000fa1e0 msgr2=0x7fa6000fc630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:11.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 -- 172.21.15.78:0/2975637562 shutdown_connections 2023-07-14T19:04:11.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.344+0000 7fa604bd3700 1 -- 172.21.15.78:0/2975637562 wait complete. 2023-07-14T19:04:11.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa604bd3700 1 Processor -- start 2023-07-14T19:04:11.353 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa604bd3700 1 -- start start 2023-07-14T19:04:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa604bd3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6000febb0 0x7fa6001958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa604bd3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 0x7fa600195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa604bd3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa600100740 0x7fa60019ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa604bd3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60019d450 con 0x7fa6000ffdb0 2023-07-14T19:04:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa604bd3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60019d590 con 0x7fa600100740 2023-07-14T19:04:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa604bd3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60019d720 con 0x7fa6000febb0 2023-07-14T19:04:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa5fdd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 0x7fa600195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:11.354 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa5fdd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 0x7fa600195e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35918/0 (socket says 172.21.15.78:35918) 2023-07-14T19:04:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa5fdd9b700 1 -- 172.21.15.78:0/3609851313 learned_addr learned my addr 172.21.15.78:0/3609851313 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.345+0000 7fa5fe59c700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6000febb0 0x7fa6001958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5fdd9b700 1 -- 172.21.15.78:0/3609851313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6000febb0 msgr2=0x7fa6001958c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5fed9d700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa600100740 0x7fa60019ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5fdd9b700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6000febb0 0x7fa6001958c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5fdd9b700 1 -- 172.21.15.78:0/3609851313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa600100740 msgr2=0x7fa60019ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:11.355 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5fdd9b700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa600100740 0x7fa60019ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5fdd9b700 1 -- 172.21.15.78:0/3609851313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f00095f0 con 0x7fa6000ffdb0 2023-07-14T19:04:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5fed9d700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa600100740 0x7fa60019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5fdd9b700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 0x7fa600195e00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00059d0 tx=0x7fa5f0003a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5f77fe700 1 -- 172.21.15.78:0/3609851313 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f001d070 con 0x7fa6000ffdb0 2023-07-14T19:04:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5f77fe700 1 -- 172.21.15.78:0/3609851313 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fa5f000bd70 con 0x7fa6000ffdb0 2023-07-14T19:04:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60019d9a0 con 0x7fa6000ffdb0 2023-07-14T19:04:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa5f77fe700 1 -- 172.21.15.78:0/3609851313 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0022c60 con 0x7fa6000ffdb0 2023-07-14T19:04:11.356 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.346+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60019ddb0 con 0x7fa6000ffdb0 2023-07-14T19:04:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.347+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa60010a220 con 0x7fa6000ffdb0 2023-07-14T19:04:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.348+0000 7fa5f77fe700 1 -- 172.21.15.78:0/3609851313 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fa5f0022470 con 0x7fa6000ffdb0 2023-07-14T19:04:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.348+0000 7fa5f77fe700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa5ec0a1150 0x7fa5ec0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.349+0000 7fa5fe59c700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa5ec0a1150 0x7fa5ec0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.349+0000 7fa5f77fe700 1 -- 172.21.15.78:0/3609851313 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fa5f00ce2d0 con 0x7fa6000ffdb0 2023-07-14T19:04:11.358 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.349+0000 7fa5fe59c700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa5ec0a1150 0x7fa5ec0a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa6000ffc10 tx=0x7fa5e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:11.359 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.353+0000 7fa5f77fe700 1 -- 172.21.15.78:0/3609851313 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fa5f00977a0 con 0x7fa6000ffdb0 2023-07-14T19:04:11.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:11 smithi088 ceph-mon[128973]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2023-07-14T19:04:11.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:11 smithi078 ceph-mon[127425]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2023-07-14T19:04:11.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.581+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fa60019e3f0 con 0x7fa6000ffdb0 2023-07-14T19:04:11.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.582+0000 7fa5f77fe700 1 -- 172.21.15.78:0/3609851313 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) v1 ==== 76+0+2884 (secure 0 0 0) 0x7fa5f009adc0 con 0x7fa6000ffdb0 2023-07-14T19:04:11.586 INFO:teuthology.orchestra.run.smithi078.stdout:e23 2023-07-14T19:04:11.586 INFO:teuthology.orchestra.run.smithi078.stdout:enable_multiple, ever_enabled_multiple: 1,1 2023-07-14T19:04:11.586 INFO:teuthology.orchestra.run.smithi078.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:04:11.587 INFO:teuthology.orchestra.run.smithi078.stdout:legacy client fscid: 1 2023-07-14T19:04:11.587 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:11.587 INFO:teuthology.orchestra.run.smithi078.stdout:Filesystem 'cephfs' (1) 2023-07-14T19:04:11.587 INFO:teuthology.orchestra.run.smithi078.stdout:fs_name cephfs 2023-07-14T19:04:11.588 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 23 2023-07-14T19:04:11.588 INFO:teuthology.orchestra.run.smithi078.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2023-07-14T19:04:11.588 INFO:teuthology.orchestra.run.smithi078.stdout:created 2023-07-14T19:02:30.340765+0000 2023-07-14T19:04:11.588 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2023-07-14T19:02:50.605732+0000 2023-07-14T19:04:11.588 INFO:teuthology.orchestra.run.smithi078.stdout:tableserver 0 2023-07-14T19:04:11.588 INFO:teuthology.orchestra.run.smithi078.stdout:root 0 2023-07-14T19:04:11.589 INFO:teuthology.orchestra.run.smithi078.stdout:session_timeout 300 2023-07-14T19:04:11.589 INFO:teuthology.orchestra.run.smithi078.stdout:session_autoclose 300 2023-07-14T19:04:11.589 INFO:teuthology.orchestra.run.smithi078.stdout:max_file_size 1099511627776 2023-07-14T19:04:11.589 INFO:teuthology.orchestra.run.smithi078.stdout:max_xattr_size 65536 2023-07-14T19:04:11.590 INFO:teuthology.orchestra.run.smithi078.stdout:required_client_features {} 2023-07-14T19:04:11.590 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure 0 2023-07-14T19:04:11.590 INFO:teuthology.orchestra.run.smithi078.stdout:last_failure_osd_epoch 0 2023-07-14T19:04:11.591 INFO:teuthology.orchestra.run.smithi078.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-07-14T19:04:11.591 INFO:teuthology.orchestra.run.smithi078.stdout:max_mds 3 2023-07-14T19:04:11.591 INFO:teuthology.orchestra.run.smithi078.stdout:in 0,1,2 2023-07-14T19:04:11.591 INFO:teuthology.orchestra.run.smithi078.stdout:up {0=24503,1=24487,2=24485} 2023-07-14T19:04:11.592 INFO:teuthology.orchestra.run.smithi078.stdout:failed 2023-07-14T19:04:11.592 INFO:teuthology.orchestra.run.smithi078.stdout:damaged 2023-07-14T19:04:11.592 INFO:teuthology.orchestra.run.smithi078.stdout:stopped 2023-07-14T19:04:11.593 INFO:teuthology.orchestra.run.smithi078.stdout:data_pools [3,4] 2023-07-14T19:04:11.593 INFO:teuthology.orchestra.run.smithi078.stdout:metadata_pool 2 2023-07-14T19:04:11.593 INFO:teuthology.orchestra.run.smithi078.stdout:inline_data disabled 2023-07-14T19:04:11.593 INFO:teuthology.orchestra.run.smithi078.stdout:balancer 2023-07-14T19:04:11.594 INFO:teuthology.orchestra.run.smithi078.stdout:bal_rank_mask -1 2023-07-14T19:04:11.594 INFO:teuthology.orchestra.run.smithi078.stdout:standby_count_wanted 0 2023-07-14T19:04:11.594 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.b{0:24503} state up:active seq 3 addr [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.594 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.c{0:24469} state up:standby-replay seq 1 addr [v2:172.21.15.102:6836/3933565644,v1:172.21.15.102:6837/3933565644] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.595 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.i{1:24487} state up:active seq 9 addr [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.595 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.e{1:24464} state up:standby-replay seq 1 addr [v2:172.21.15.88:6834/3667124043,v1:172.21.15.88:6835/3667124043] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.595 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.h{2:24485} state up:active seq 10 addr [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.596 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.f{2:24457} state up:standby-replay seq 1 addr [v2:172.21.15.102:6834/75062394,v1:172.21.15.102:6835/75062394] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:11.596 INFO:teuthology.orchestra.run.smithi078.stdout:Standby daemons: 2023-07-14T19:04:11.596 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:11.597 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.d{-1:14559} state up:standby seq 1 addr [v2:172.21.15.78:6836/4055842453,v1:172.21.15.78:6837/4055842453] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.597 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.g{-1:14616} state up:standby seq 1 addr [v2:172.21.15.78:6838/680498429,v1:172.21.15.78:6839/680498429] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.597 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.a{-1:14628} state up:standby seq 1 addr [v2:172.21.15.78:6840/1070208,v1:172.21.15.78:6841/1070208] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.597 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.l{-1:24418} state up:standby seq 1 addr [v2:172.21.15.102:6832/2462741496,v1:172.21.15.102:6833/2462741496] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.597 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.k{-1:24428} state up:standby seq 1 addr [v2:172.21.15.88:6832/244273805,v1:172.21.15.88:6833/244273805] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.597 INFO:teuthology.orchestra.run.smithi078.stdout:[mds.j{-1:24440} state up:standby seq 1 addr [v2:172.21.15.78:6834/2843510081,v1:172.21.15.78:6835/2843510081] compat {c=[1],r=[1],i=[7ff]}] 2023-07-14T19:04:11.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa5ec0a1150 msgr2=0x7fa5ec0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:11.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa5ec0a1150 0x7fa5ec0a3600 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fa6000ffc10 tx=0x7fa5e800b410 comp rx=0 tx=0).stop 2023-07-14T19:04:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 msgr2=0x7fa600195e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 0x7fa600195e00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fa5f00059d0 tx=0x7fa5f0003a50 comp rx=0 tx=0).stop 2023-07-14T19:04:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 shutdown_connections 2023-07-14T19:04:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fa5ec0a1150 0x7fa5ec0a3600 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fa600100740 0x7fa60019ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa6000febb0 0x7fa6001958c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.599 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 --2- 172.21.15.78:0/3609851313 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fa6000ffdb0 0x7fa600195e00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 >> 172.21.15.78:0/3609851313 conn(0x7fa6000fa1e0 msgr2=0x7fa600068100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 shutdown_connections 2023-07-14T19:04:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:11.584+0000 7fa604bd3700 1 -- 172.21.15.78:0/3609851313 wait complete. 2023-07-14T19:04:11.600 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2023-07-14T19:04:11.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:11 smithi102 ceph-mon[129280]: pgmap v307: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 7 op/s 2023-07-14T19:04:12.060 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- bash -c 'ceph osd dump' 2023-07-14T19:04:12.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:12 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3609851313' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:04:12.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:12 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3609851313' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:04:12.612 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:04:12.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:12 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3609851313' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2023-07-14T19:04:13.397 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:13 smithi078 ceph-mon[127425]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:13.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:13 smithi088 ceph-mon[128973]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:13.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:13 smithi102 ceph-mon[129280]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 6 op/s 2023-07-14T19:04:13.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 -- 172.21.15.78:0/1954611473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500fff20 msgr2=0x7f7c501067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:13.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 --2- 172.21.15.78:0/1954611473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500fff20 0x7f7c501067e0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7c4800b0a0 tx=0x7f7c4800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:13.924 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 -- 172.21.15.78:0/1954611473 shutdown_connections 2023-07-14T19:04:13.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 --2- 172.21.15.78:0/1954611473 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c500fe390 0x7f7c500fe7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:13.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 --2- 172.21.15.78:0/1954611473 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500fff20 0x7f7c501067e0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:13.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 --2- 172.21.15.78:0/1954611473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c500ff590 0x7f7c500ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 -- 172.21.15.78:0/1954611473 >> 172.21.15.78:0/1954611473 conn(0x7f7c500f99c0 msgr2=0x7f7c500fbe10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 -- 172.21.15.78:0/1954611473 shutdown_connections 2023-07-14T19:04:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 -- 172.21.15.78:0/1954611473 wait complete. 2023-07-14T19:04:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 Processor -- start 2023-07-14T19:04:13.926 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.918+0000 7f7c54820700 1 -- start start 2023-07-14T19:04:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c54820700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c500fe390 0x7f7c501025e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c54820700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500ff590 0x7f7c50100c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c54820700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c500fff20 0x7f7c50101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c54820700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c501018f0 con 0x7f7c500ff590 2023-07-14T19:04:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c54820700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c50101a60 con 0x7f7c500fff20 2023-07-14T19:04:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c54820700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c5019faa0 con 0x7f7c500fe390 2023-07-14T19:04:13.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500ff590 0x7f7c50100c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500ff590 0x7f7c50100c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35928/0 (socket says 172.21.15.78:35928) 2023-07-14T19:04:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 -- 172.21.15.78:0/2913423237 learned_addr learned my addr 172.21.15.78:0/2913423237 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4e59c700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c500fe390 0x7f7c501025e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4ed9d700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c500fff20 0x7f7c50101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 -- 172.21.15.78:0/2913423237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c500fe390 msgr2=0x7f7c501025e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c500fe390 0x7f7c501025e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:13.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 -- 172.21.15.78:0/2913423237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c500fff20 msgr2=0x7f7c50101170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c500fff20 0x7f7c50101170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 -- 172.21.15.78:0/2913423237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c48009d00 con 0x7f7c500ff590 2023-07-14T19:04:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c4dd9b700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500ff590 0x7f7c50100c30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400ddc0 tx=0x7f7c4400a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c3f7fe700 1 -- 172.21.15.78:0/2913423237 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4400dea0 con 0x7f7c500ff590 2023-07-14T19:04:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c5019fd80 con 0x7f7c500ff590 2023-07-14T19:04:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c3f7fe700 1 -- 172.21.15.78:0/2913423237 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7c4400e8e0 con 0x7f7c500ff590 2023-07-14T19:04:13.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.919+0000 7f7c3f7fe700 1 -- 172.21.15.78:0/2913423237 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c44016860 con 0x7f7c500ff590 2023-07-14T19:04:13.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.920+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c501a0270 con 0x7f7c500ff590 2023-07-14T19:04:13.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.920+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c501091d0 con 0x7f7c500ff590 2023-07-14T19:04:13.930 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.921+0000 7f7c3f7fe700 1 -- 172.21.15.78:0/2913423237 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f7c4400e450 con 0x7f7c500ff590 2023-07-14T19:04:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.922+0000 7f7c3f7fe700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7c380a1150 0x7f7c380a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.922+0000 7f7c4e59c700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7c380a1150 0x7f7c380a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.922+0000 7f7c3f7fe700 1 -- 172.21.15.78:0/2913423237 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f7c440ccb40 con 0x7f7c500ff590 2023-07-14T19:04:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.922+0000 7f7c4e59c700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7c380a1150 0x7f7c380a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7c40005fd0 tx=0x7f7c40005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:13.931 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:13.926+0000 7f7c3f7fe700 1 -- 172.21.15.78:0/2913423237 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7c44095f80 con 0x7f7c500ff590 2023-07-14T19:04:14.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.113+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f7c501093c0 con 0x7f7c500ff590 2023-07-14T19:04:14.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.114+0000 7f7c3f7fe700 1 -- 172.21.15.78:0/2913423237 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v87) v1 ==== 56+0+5591 (secure 0 0 0) 0x7f7c440995a0 con 0x7f7c500ff590 2023-07-14T19:04:14.121 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 87 2023-07-14T19:04:14.121 INFO:teuthology.orchestra.run.smithi078.stdout:fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T19:04:14.121 INFO:teuthology.orchestra.run.smithi078.stdout:created 2023-07-14T18:55:36.239751+0000 2023-07-14T19:04:14.121 INFO:teuthology.orchestra.run.smithi078.stdout:modified 2023-07-14T19:03:31.201384+0000 2023-07-14T19:04:14.122 INFO:teuthology.orchestra.run.smithi078.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2023-07-14T19:04:14.122 INFO:teuthology.orchestra.run.smithi078.stdout:crush_version 27 2023-07-14T19:04:14.122 INFO:teuthology.orchestra.run.smithi078.stdout:full_ratio 0.95 2023-07-14T19:04:14.122 INFO:teuthology.orchestra.run.smithi078.stdout:backfillfull_ratio 0.9 2023-07-14T19:04:14.122 INFO:teuthology.orchestra.run.smithi078.stdout:nearfull_ratio 0.85 2023-07-14T19:04:14.122 INFO:teuthology.orchestra.run.smithi078.stdout:require_min_compat_client luminous 2023-07-14T19:04:14.122 INFO:teuthology.orchestra.run.smithi078.stdout:min_compat_client luminous 2023-07-14T19:04:14.123 INFO:teuthology.orchestra.run.smithi078.stdout:require_osd_release reef 2023-07-14T19:04:14.123 INFO:teuthology.orchestra.run.smithi078.stdout:stretch_mode_enabled false 2023-07-14T19:04:14.123 INFO:teuthology.orchestra.run.smithi078.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 22 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2023-07-14T19:04:14.123 INFO:teuthology.orchestra.run.smithi078.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 80 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2023-07-14T19:04:14.123 INFO:teuthology.orchestra.run.smithi078.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 81 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2023-07-14T19:04:14.123 INFO:teuthology.orchestra.run.smithi078.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2023-07-14T19:04:14.124 INFO:teuthology.orchestra.run.smithi078.stdout:max_osd 12 2023-07-14T19:04:14.124 INFO:teuthology.orchestra.run.smithi078.stdout:osd.0 up in weight 1 up_from 8 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6802/1630210341,v1:172.21.15.78:6803/1630210341] [v2:172.21.15.78:6804/1630210341,v1:172.21.15.78:6805/1630210341] exists,up 94b51247-9012-4bba-941b-2a524896715f 2023-07-14T19:04:14.124 INFO:teuthology.orchestra.run.smithi078.stdout:osd.1 up in weight 1 up_from 13 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6800/3959843234,v1:172.21.15.88:6801/3959843234] [v2:172.21.15.88:6802/3959843234,v1:172.21.15.88:6803/3959843234] exists,up e5f66a20-1101-4355-8ca4-9d3de118d5c8 2023-07-14T19:04:14.124 INFO:teuthology.orchestra.run.smithi078.stdout:osd.2 up in weight 1 up_from 18 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6800/861816843,v1:172.21.15.102:6801/861816843] [v2:172.21.15.102:6802/861816843,v1:172.21.15.102:6803/861816843] exists,up 180e6f44-2edd-427f-ac80-850d9e605f23 2023-07-14T19:04:14.124 INFO:teuthology.orchestra.run.smithi078.stdout:osd.3 up in weight 1 up_from 25 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6810/4191618186,v1:172.21.15.78:6811/4191618186] [v2:172.21.15.78:6812/4191618186,v1:172.21.15.78:6813/4191618186] exists,up 2e708d95-c299-4868-bbea-8ac0241d4118 2023-07-14T19:04:14.124 INFO:teuthology.orchestra.run.smithi078.stdout:osd.4 up in weight 1 up_from 33 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6808/1708042388,v1:172.21.15.88:6809/1708042388] [v2:172.21.15.88:6810/1708042388,v1:172.21.15.88:6811/1708042388] exists,up f2a8821e-94f2-4927-a1c6-c1bcce365fea 2023-07-14T19:04:14.124 INFO:teuthology.orchestra.run.smithi078.stdout:osd.5 up in weight 1 up_from 38 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6808/3895937355,v1:172.21.15.102:6809/3895937355] [v2:172.21.15.102:6810/3895937355,v1:172.21.15.102:6811/3895937355] exists,up 7abc51f7-2397-4f2a-8046-0b93c4a56115 2023-07-14T19:04:14.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.6 up in weight 1 up_from 45 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6818/2536976719,v1:172.21.15.78:6819/2536976719] [v2:172.21.15.78:6820/2536976719,v1:172.21.15.78:6821/2536976719] exists,up c02e47de-7c3a-41c9-bb4c-d339519b6a06 2023-07-14T19:04:14.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.7 up in weight 1 up_from 50 up_thru 83 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6816/1000622113,v1:172.21.15.88:6817/1000622113] [v2:172.21.15.88:6818/1000622113,v1:172.21.15.88:6819/1000622113] exists,up 603c8010-0bcc-4cb4-9e64-308839b69ea1 2023-07-14T19:04:14.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.8 up in weight 1 up_from 57 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6816/4258575075,v1:172.21.15.102:6817/4258575075] [v2:172.21.15.102:6818/4258575075,v1:172.21.15.102:6819/4258575075] exists,up 560f9ecc-e1d9-472d-af8b-ac1575902103 2023-07-14T19:04:14.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.9 up in weight 1 up_from 64 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.78:6826/1815793298,v1:172.21.15.78:6827/1815793298] [v2:172.21.15.78:6828/1815793298,v1:172.21.15.78:6829/1815793298] exists,up 8afa084a-aebe-4fbf-af3b-36fcbf682dc4 2023-07-14T19:04:14.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.10 up in weight 1 up_from 69 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.88:6824/2764459881,v1:172.21.15.88:6825/2764459881] [v2:172.21.15.88:6826/2764459881,v1:172.21.15.88:6827/2764459881] exists,up f150df7b-04f5-4103-bd4f-d93f48408f48 2023-07-14T19:04:14.125 INFO:teuthology.orchestra.run.smithi078.stdout:osd.11 up in weight 1 up_from 76 up_thru 86 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6824/383662404,v1:172.21.15.102:6825/383662404] [v2:172.21.15.102:6826/383662404,v1:172.21.15.102:6827/383662404] exists,up dc848771-b0b4-4e08-838a-25775af1cefb 2023-07-14T19:04:14.125 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.10 [8,11] 2023-07-14T19:04:14.126 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.13 [2,11] 2023-07-14T19:04:14.126 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.1a [8,1] 2023-07-14T19:04:14.126 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 2.1d [2,4] 2023-07-14T19:04:14.126 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.11 [4,8] 2023-07-14T19:04:14.126 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 3.36 [4,9] 2023-07-14T19:04:14.126 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.f [9,10] 2023-07-14T19:04:14.126 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.12 [5,10] 2023-07-14T19:04:14.127 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.16 [9,8] 2023-07-14T19:04:14.127 INFO:teuthology.orchestra.run.smithi078.stdout:pg_upmap_items 4.18 [9,8] 2023-07-14T19:04:14.127 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/378103227 expires 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:14.127 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1800545696 expires 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:14.127 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6800/3249522727 expires 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:14.127 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/378103227 expires 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:14.127 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1797764467 expires 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:14.127 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:6801/3249522727 expires 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:14.128 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/21455120 expires 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:14.128 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/3838414629 expires 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:14.128 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/1459224377 expires 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:14.128 INFO:teuthology.orchestra.run.smithi078.stdout:blocklist 172.21.15.78:0/2696766201 expires 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.115+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7c380a1150 msgr2=0x7f7c380a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.115+0000 7f7c54820700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7c380a1150 0x7f7c380a3600 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f7c40005fd0 tx=0x7f7c40005dd0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.115+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500ff590 msgr2=0x7f7c50100c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.115+0000 7f7c54820700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500ff590 0x7f7c50100c30 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400ddc0 tx=0x7f7c4400a530 comp rx=0 tx=0).stop 2023-07-14T19:04:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.116+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 shutdown_connections 2023-07-14T19:04:14.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.116+0000 7f7c54820700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f7c380a1150 0x7f7c380a3600 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.116+0000 7f7c54820700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7c500fff20 0x7f7c50101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.116+0000 7f7c54820700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7c500fe390 0x7f7c501025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.116+0000 7f7c54820700 1 --2- 172.21.15.78:0/2913423237 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7c500ff590 0x7f7c50100c30 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.116+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 >> 172.21.15.78:0/2913423237 conn(0x7f7c500f99c0 msgr2=0x7f7c50104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.116+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 shutdown_connections 2023-07-14T19:04:14.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.116+0000 7f7c54820700 1 -- 172.21.15.78:0/2913423237 wait complete. 2023-07-14T19:04:14.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:14 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2913423237' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:04:14.504 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:14 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2913423237' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:04:14.561 INFO:teuthology.run_tasks:Running task fs.ready... 2023-07-14T19:04:14.578 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:04:14.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:14 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2913423237' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2023-07-14T19:04:14.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.701+0000 7f6c87848700 1 -- 172.21.15.78:0/2483703537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c80100690 msgr2=0x7f6c80106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:14.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.701+0000 7f6c87848700 1 --2- 172.21.15.78:0/2483703537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c80100690 0x7f6c80106f50 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f6c7800b0a0 tx=0x7f6c7800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.701+0000 7f6c87848700 1 -- 172.21.15.78:0/2483703537 shutdown_connections 2023-07-14T19:04:14.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.701+0000 7f6c87848700 1 --2- 172.21.15.78:0/2483703537 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c80100690 0x7f6c80106f50 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.701+0000 7f6c87848700 1 --2- 172.21.15.78:0/2483703537 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c800feb10 0x7f6c800fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.701+0000 7f6c87848700 1 --2- 172.21.15.78:0/2483703537 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c800ffd00 0x7f6c80100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.701+0000 7f6c87848700 1 -- 172.21.15.78:0/2483703537 >> 172.21.15.78:0/2483703537 conn(0x7f6c800fa130 msgr2=0x7f6c800fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:14.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 -- 172.21.15.78:0/2483703537 shutdown_connections 2023-07-14T19:04:14.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 -- 172.21.15.78:0/2483703537 wait complete. 2023-07-14T19:04:14.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 Processor -- start 2023-07-14T19:04:14.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 -- start start 2023-07-14T19:04:14.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c800feb10 0x7f6c80069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:14.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c800ffd00 0x7f6c8019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:14.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c80100690 0x7f6c8019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:14.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8019d4d0 con 0x7f6c800feb10 2023-07-14T19:04:14.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8019d640 con 0x7f6c800ffd00 2023-07-14T19:04:14.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c87848700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8019d800 con 0x7f6c80100690 2023-07-14T19:04:14.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c855e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c800feb10 0x7f6c80069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:14.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.702+0000 7f6c855e4700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c800feb10 0x7f6c80069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35940/0 (socket says 172.21.15.78:35940) 2023-07-14T19:04:14.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c855e4700 1 -- 172.21.15.78:0/1623015333 learned_addr learned my addr 172.21.15.78:0/1623015333 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:14.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c84de3700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c800ffd00 0x7f6c8019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:14.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c85de5700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c80100690 0x7f6c8019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:14.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c855e4700 1 -- 172.21.15.78:0/1623015333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c80100690 msgr2=0x7f6c8019cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:14.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c855e4700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c80100690 0x7f6c8019cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c855e4700 1 -- 172.21.15.78:0/1623015333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c800ffd00 msgr2=0x7f6c8019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:14.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c855e4700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c800ffd00 0x7f6c8019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c855e4700 1 -- 172.21.15.78:0/1623015333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c78009d00 con 0x7f6c800feb10 2023-07-14T19:04:14.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c855e4700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c800feb10 0x7f6c80069530 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400b500 tx=0x7f6c7400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:14.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c727fc700 1 -- 172.21.15.78:0/1623015333 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74011830 con 0x7f6c800feb10 2023-07-14T19:04:14.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c727fc700 1 -- 172.21.15.78:0/1623015333 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6c74011e20 con 0x7f6c800feb10 2023-07-14T19:04:14.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8019dae0 con 0x7f6c800feb10 2023-07-14T19:04:14.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c727fc700 1 -- 172.21.15.78:0/1623015333 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c74010dd0 con 0x7f6c800feb10 2023-07-14T19:04:14.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.703+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8019dff0 con 0x7f6c800feb10 2023-07-14T19:04:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.704+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8010a230 con 0x7f6c800feb10 2023-07-14T19:04:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.705+0000 7f6c727fc700 1 -- 172.21.15.78:0/1623015333 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f6c74011990 con 0x7f6c800feb10 2023-07-14T19:04:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.705+0000 7f6c727fc700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6c6c0a1220 0x7f6c6c0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.706+0000 7f6c84de3700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6c6c0a1220 0x7f6c6c0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:14.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.706+0000 7f6c727fc700 1 -- 172.21.15.78:0/1623015333 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f6c740ccb60 con 0x7f6c800feb10 2023-07-14T19:04:14.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.706+0000 7f6c84de3700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6c6c0a1220 0x7f6c6c0a36d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c005950 tx=0x7f6c7c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:14.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.709+0000 7f6c727fc700 1 -- 172.21.15.78:0/1623015333 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f6c74095f40 con 0x7f6c800feb10 2023-07-14T19:04:14.908 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.904+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c80105980 con 0x7f6c800feb10 2023-07-14T19:04:14.909 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.906+0000 7f6c727fc700 1 -- 172.21.15.78:0/1623015333 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24453 (secure 0 0 0) 0x7f6c74099560 con 0x7f6c800feb10 2023-07-14T19:04:14.910 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:14.910 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:03:31.201384+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:04:14.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6c6c0a1220 msgr2=0x7f6c6c0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:14.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6c6c0a1220 0x7f6c6c0a36d0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c005950 tx=0x7f6c7c0058e0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c800feb10 msgr2=0x7f6c80069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:14.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c800feb10 0x7f6c80069530 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f6c7400b500 tx=0x7f6c7400b8c0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 shutdown_connections 2023-07-14T19:04:14.912 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f6c6c0a1220 0x7f6c6c0a36d0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c80100690 0x7f6c8019cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f6c800feb10 0x7f6c80069530 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 --2- 172.21.15.78:0/1623015333 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f6c800ffd00 0x7f6c8019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:14.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 >> 172.21.15.78:0/1623015333 conn(0x7f6c800fa130 msgr2=0x7f6c800fc300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:14.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 shutdown_connections 2023-07-14T19:04:14.913 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:14.908+0000 7f6c87848700 1 -- 172.21.15.78:0/1623015333 wait complete. 2023-07-14T19:04:14.924 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:02:28.269050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:02:29.547771+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T19:02:32.797605+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T19:04:14.925 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:04:15.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 -- 172.21.15.78:0/436482912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0feb10 msgr2=0x7f3e3c0fef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 --2- 172.21.15.78:0/436482912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0feb10 0x7f3e3c0fef20 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009990 tx=0x7f3e38009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 -- 172.21.15.78:0/436482912 shutdown_connections 2023-07-14T19:04:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 --2- 172.21.15.78:0/436482912 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e3c100690 0x7f3e3c106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 --2- 172.21.15.78:0/436482912 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0feb10 0x7f3e3c0fef20 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 --2- 172.21.15.78:0/436482912 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3c0ffd00 0x7f3e3c100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 -- 172.21.15.78:0/436482912 >> 172.21.15.78:0/436482912 conn(0x7f3e3c0fa130 msgr2=0x7f3e3c0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 -- 172.21.15.78:0/436482912 shutdown_connections 2023-07-14T19:04:15.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 -- 172.21.15.78:0/436482912 wait complete. 2023-07-14T19:04:15.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 Processor -- start 2023-07-14T19:04:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.047+0000 7f3e445a6700 1 -- start start 2023-07-14T19:04:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e445a6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3c0feb10 0x7f3e3c068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e445a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0ffd00 0x7f3e3c19ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e445a6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e3c100690 0x7f3e3c19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e445a6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c19f600 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e445a6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c19f770 con 0x7f3e3c100690 2023-07-14T19:04:15.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e445a6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3c19f930 con 0x7f3e3c0feb10 2023-07-14T19:04:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0ffd00 0x7f3e3c19ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0ffd00 0x7f3e3c19ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35956/0 (socket says 172.21.15.78:35956) 2023-07-14T19:04:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 -- 172.21.15.78:0/1285872436 learned_addr learned my addr 172.21.15.78:0/1285872436 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e42b43700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e3c100690 0x7f3e3c19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e42342700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3c0feb10 0x7f3e3c068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 -- 172.21.15.78:0/1285872436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3c0feb10 msgr2=0x7f3e3c068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3c0feb10 0x7f3e3c068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 -- 172.21.15.78:0/1285872436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e3c100690 msgr2=0x7f3e3c19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e3c100690 0x7f3e3c19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 -- 172.21.15.78:0/1285872436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380095f0 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e41b41700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0ffd00 0x7f3e3c19ca00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00d950 tx=0x7f3e2c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e337fe700 1 -- 172.21.15.78:0/1285872436 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c009b40 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e337fe700 1 -- 172.21.15.78:0/1285872436 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3e2c009ca0 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3c19fc10 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e337fe700 1 -- 172.21.15.78:0/1285872436 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c00f620 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.048+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3c1a0170 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.049+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e3c105b20 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.050+0000 7f3e337fe700 1 -- 172.21.15.78:0/1285872436 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3e2c010460 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.051+0000 7f3e337fe700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e280a11d0 0x7f3e280a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.051+0000 7f3e337fe700 1 -- 172.21.15.78:0/1285872436 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3e2c0ccf30 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.051+0000 7f3e42342700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e280a11d0 0x7f3e280a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.051+0000 7f3e42342700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e280a11d0 0x7f3e280a3680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3e3800c010 tx=0x7f3e3800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:15.060 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.054+0000 7f3e337fe700 1 -- 172.21.15.78:0/1285872436 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3e2c096370 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.244+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3e3c02cb30 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.244+0000 7f3e337fe700 1 -- 172.21.15.78:0/1285872436 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3e2c016020 con 0x7f3e3c0ffd00 2023-07-14T19:04:15.249 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:04:15.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e280a11d0 msgr2=0x7f3e280a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e280a11d0 0x7f3e280a3680 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3e3800c010 tx=0x7f3e3800b540 comp rx=0 tx=0).stop 2023-07-14T19:04:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0ffd00 msgr2=0x7f3e3c19ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0ffd00 0x7f3e3c19ca00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00d950 tx=0x7f3e2c00dd10 comp rx=0 tx=0).stop 2023-07-14T19:04:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 shutdown_connections 2023-07-14T19:04:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e280a11d0 0x7f3e280a3680 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e3c100690 0x7f3e3c19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e3c0feb10 0x7f3e3c068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 --2- 172.21.15.78:0/1285872436 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e3c0ffd00 0x7f3e3c19ca00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 >> 172.21.15.78:0/1285872436 conn(0x7f3e3c0fa130 msgr2=0x7f3e3c0fc470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:15.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 shutdown_connections 2023-07-14T19:04:15.254 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.247+0000 7f3e445a6700 1 -- 172.21.15.78:0/1285872436 wait complete. 2023-07-14T19:04:15.262 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:04:15.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 -- 172.21.15.78:0/95075343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 msgr2=0x7f8608100520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 --2- 172.21.15.78:0/95075343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 0x7f8608100520 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f85f8009990 tx=0x7f85f8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 -- 172.21.15.78:0/95075343 shutdown_connections 2023-07-14T19:04:15.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 --2- 172.21.15.78:0/95075343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86080ff710 0x7f8608106fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 --2- 172.21.15.78:0/95075343 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 0x7f8608100520 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 --2- 172.21.15.78:0/95075343 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86080fed60 0x7f86080ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 -- 172.21.15.78:0/95075343 >> 172.21.15.78:0/95075343 conn(0x7f86080f98e0 msgr2=0x7f86080fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:15.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 -- 172.21.15.78:0/95075343 shutdown_connections 2023-07-14T19:04:15.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.384+0000 7f860d776700 1 -- 172.21.15.78:0/95075343 wait complete. 2023-07-14T19:04:15.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f860d776700 1 Processor -- start 2023-07-14T19:04:15.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f860d776700 1 -- start start 2023-07-14T19:04:15.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f860d776700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86080fed60 0x7f8608194740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f860d776700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86080ff710 0x7f8608194c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f860d776700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 0x7f860819bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f860d776700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860819c2c0 con 0x7f8608100110 2023-07-14T19:04:15.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f860d776700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860819c400 con 0x7f86080fed60 2023-07-14T19:04:15.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f860d776700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860819c590 con 0x7f86080ff710 2023-07-14T19:04:15.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f86077fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 0x7f860819bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f86077fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 0x7f860819bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35970/0 (socket says 172.21.15.78:35970) 2023-07-14T19:04:15.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.385+0000 7f86077fe700 1 -- 172.21.15.78:0/2915099773 learned_addr learned my addr 172.21.15.78:0/2915099773 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:15.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f86067fc700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86080ff710 0x7f8608194c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f8606ffd700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86080fed60 0x7f8608194740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f86077fe700 1 -- 172.21.15.78:0/2915099773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86080ff710 msgr2=0x7f8608194c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f86077fe700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86080ff710 0x7f8608194c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f86077fe700 1 -- 172.21.15.78:0/2915099773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86080fed60 msgr2=0x7f8608194740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f86077fe700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86080fed60 0x7f8608194740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f86077fe700 1 -- 172.21.15.78:0/2915099773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f80095f0 con 0x7f8608100110 2023-07-14T19:04:15.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f8606ffd700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86080fed60 0x7f8608194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:15.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f86077fe700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 0x7f860819bcf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8600007ee0 tx=0x7f860000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:15.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f85fffff700 1 -- 172.21.15.78:0/2915099773 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600017070 con 0x7f8608100110 2023-07-14T19:04:15.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f85fffff700 1 -- 172.21.15.78:0/2915099773 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f860000f040 con 0x7f8608100110 2023-07-14T19:04:15.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860819c870 con 0x7f8608100110 2023-07-14T19:04:15.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f85fffff700 1 -- 172.21.15.78:0/2915099773 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8600005710 con 0x7f8608100110 2023-07-14T19:04:15.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.386+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860819cd30 con 0x7f8608100110 2023-07-14T19:04:15.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.387+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8608105ab0 con 0x7f8608100110 2023-07-14T19:04:15.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.388+0000 7f85fffff700 1 -- 172.21.15.78:0/2915099773 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8600005870 con 0x7f8608100110 2023-07-14T19:04:15.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.389+0000 7f85fffff700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85f40a1150 0x7f85f40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.389+0000 7f8606ffd700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85f40a1150 0x7f85f40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.389+0000 7f85fffff700 1 -- 172.21.15.78:0/2915099773 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f86000cc490 con 0x7f8608100110 2023-07-14T19:04:15.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.389+0000 7f8606ffd700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85f40a1150 0x7f85f40a3600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f85f800b5c0 tx=0x7f85f8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:15.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.393+0000 7f85fffff700 1 -- 172.21.15.78:0/2915099773 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8600095870 con 0x7f8608100110 2023-07-14T19:04:15.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:15 smithi078 ceph-mon[127425]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:04:15.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:15 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1623015333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:15.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.585+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8608061280 con 0x7f8608100110 2023-07-14T19:04:15.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.585+0000 7f85fffff700 1 -- 172.21.15.78:0/2915099773 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7f8600098e90 con 0x7f8608100110 2023-07-14T19:04:15.589 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:04:15.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85f40a1150 msgr2=0x7f85f40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85f40a1150 0x7f85f40a3600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f85f800b5c0 tx=0x7f85f8000bc0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 msgr2=0x7f860819bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 0x7f860819bcf0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f8600007ee0 tx=0x7f860000d3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 shutdown_connections 2023-07-14T19:04:15.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85f40a1150 0x7f85f40a3600 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f86080fed60 0x7f8608194740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8608100110 0x7f860819bcf0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 --2- 172.21.15.78:0/2915099773 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f86080ff710 0x7f8608194c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 >> 172.21.15.78:0/2915099773 conn(0x7f86080f98e0 msgr2=0x7f86080fbb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:15.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 shutdown_connections 2023-07-14T19:04:15.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.587+0000 7f860d776700 1 -- 172.21.15.78:0/2915099773 wait complete. 2023-07-14T19:04:15.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:04:15.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:15 smithi102 ceph-mon[129280]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:04:15.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:15 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1623015333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:15.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:15 smithi088 ceph-mon[128973]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 6 op/s 2023-07-14T19:04:15.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:15 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1623015333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:15.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 -- 172.21.15.78:0/432559274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e0100690 msgr2=0x7f97e0106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 --2- 172.21.15.78:0/432559274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e0100690 0x7f97e0106f50 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f97d800b0a0 tx=0x7f97d800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 -- 172.21.15.78:0/432559274 shutdown_connections 2023-07-14T19:04:15.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 --2- 172.21.15.78:0/432559274 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97e00feb10 0x7f97e00fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 --2- 172.21.15.78:0/432559274 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e0100690 0x7f97e0106f50 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 --2- 172.21.15.78:0/432559274 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e00ffd00 0x7f97e0100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 -- 172.21.15.78:0/432559274 >> 172.21.15.78:0/432559274 conn(0x7f97e00fa130 msgr2=0x7f97e00fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:15.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 -- 172.21.15.78:0/432559274 shutdown_connections 2023-07-14T19:04:15.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.724+0000 7f97e74a6700 1 -- 172.21.15.78:0/432559274 wait complete. 2023-07-14T19:04:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e74a6700 1 Processor -- start 2023-07-14T19:04:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e74a6700 1 -- start start 2023-07-14T19:04:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e74a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e00feb10 0x7f97e0102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e74a6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e00ffd00 0x7f97e01010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e74a6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97e0100690 0x7f97e0101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e74a6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0101dc0 con 0x7f97e00feb10 2023-07-14T19:04:15.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e74a6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e0101f30 con 0x7f97e0100690 2023-07-14T19:04:15.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e74a6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97e019fa00 con 0x7f97e00ffd00 2023-07-14T19:04:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e5242700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e00feb10 0x7f97e0102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e5242700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e00feb10 0x7f97e0102aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35980/0 (socket says 172.21.15.78:35980) 2023-07-14T19:04:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e5242700 1 -- 172.21.15.78:0/3502735711 learned_addr learned my addr 172.21.15.78:0/3502735711 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:15.736 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e4a41700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e00ffd00 0x7f97e01010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.725+0000 7f97e5a43700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97e0100690 0x7f97e0101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97e5242700 1 -- 172.21.15.78:0/3502735711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e00ffd00 msgr2=0x7f97e01010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97e5242700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e00ffd00 0x7f97e01010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97e5242700 1 -- 172.21.15.78:0/3502735711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97e0100690 msgr2=0x7f97e0101620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97e5242700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97e0100690 0x7f97e0101620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.737 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97e5242700 1 -- 172.21.15.78:0/3502735711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97d8009d00 con 0x7f97e00feb10 2023-07-14T19:04:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97e5242700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e00feb10 0x7f97e0102aa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f97d000d980 tx=0x7f97d000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97d67fc700 1 -- 172.21.15.78:0/3502735711 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d0004020 con 0x7f97e00feb10 2023-07-14T19:04:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97d67fc700 1 -- 172.21.15.78:0/3502735711 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f97d000f690 con 0x7f97e00feb10 2023-07-14T19:04:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97e019fce0 con 0x7f97e00feb10 2023-07-14T19:04:15.738 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97d67fc700 1 -- 172.21.15.78:0/3502735711 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97d00039e0 con 0x7f97e00feb10 2023-07-14T19:04:15.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.726+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97e01a01a0 con 0x7f97e00feb10 2023-07-14T19:04:15.739 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.727+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97e010a1a0 con 0x7f97e00feb10 2023-07-14T19:04:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.728+0000 7f97d67fc700 1 -- 172.21.15.78:0/3502735711 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f97d000f800 con 0x7f97e00feb10 2023-07-14T19:04:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.729+0000 7f97d67fc700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f97cc0a1150 0x7f97cc0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.729+0000 7f97e4a41700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f97cc0a1150 0x7f97cc0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.729+0000 7f97d67fc700 1 -- 172.21.15.78:0/3502735711 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f97d00ccb60 con 0x7f97e00feb10 2023-07-14T19:04:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.729+0000 7f97e4a41700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f97cc0a1150 0x7f97cc0a3600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f97dc005950 tx=0x7f97dc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:15.740 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.732+0000 7f97d67fc700 1 -- 172.21.15.78:0/3502735711 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f97d0095fa0 con 0x7f97e00feb10 2023-07-14T19:04:15.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.921+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f97e01a0750 con 0x7f97e00feb10 2023-07-14T19:04:15.925 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.922+0000 7f97d67fc700 1 -- 172.21.15.78:0/3502735711 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f97d00995c0 con 0x7f97e00feb10 2023-07-14T19:04:15.925 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-07-14T19:04:15.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f97cc0a1150 msgr2=0x7f97cc0a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.927 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f97cc0a1150 0x7f97cc0a3600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f97dc005950 tx=0x7f97dc00b410 comp rx=0 tx=0).stop 2023-07-14T19:04:15.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e00feb10 msgr2=0x7f97e0102aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:15.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e00feb10 0x7f97e0102aa0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f97d000d980 tx=0x7f97d000dc90 comp rx=0 tx=0).stop 2023-07-14T19:04:15.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 shutdown_connections 2023-07-14T19:04:15.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f97cc0a1150 0x7f97cc0a3600 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f97e0100690 0x7f97e0101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f97e00feb10 0x7f97e0102aa0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.928 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 --2- 172.21.15.78:0/3502735711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f97e00ffd00 0x7f97e01010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:15.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 >> 172.21.15.78:0/3502735711 conn(0x7f97e00fa130 msgr2=0x7f97e0067860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:15.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 shutdown_connections 2023-07-14T19:04:15.929 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:15.924+0000 7f97e74a6700 1 -- 172.21.15.78:0/3502735711 wait complete. 2023-07-14T19:04:15.940 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T19:04:16.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 -- 172.21.15.78:0/524920667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad0100770 msgr2=0x7f8ad0107010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.065 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 --2- 172.21.15.78:0/524920667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad0100770 0x7f8ad0107010 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f8ac800b0a0 tx=0x7f8ac800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 -- 172.21.15.78:0/524920667 shutdown_connections 2023-07-14T19:04:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 --2- 172.21.15.78:0/524920667 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ad00feb30 0x7f8ad00fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 --2- 172.21.15.78:0/524920667 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad0100770 0x7f8ad0107010 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 --2- 172.21.15.78:0/524920667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad00ffdc0 0x7f8ad0100230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.066 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 -- 172.21.15.78:0/524920667 >> 172.21.15.78:0/524920667 conn(0x7f8ad00fa150 msgr2=0x7f8ad00fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:16.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 -- 172.21.15.78:0/524920667 shutdown_connections 2023-07-14T19:04:16.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.061+0000 7f8ad6963700 1 -- 172.21.15.78:0/524920667 wait complete. 2023-07-14T19:04:16.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8ad6963700 1 Processor -- start 2023-07-14T19:04:16.067 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8ad6963700 1 -- start start 2023-07-14T19:04:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8ad6963700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad00feb30 0x7f8ad0194850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8ad6963700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad00ffdc0 0x7f8ad0194d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8ad6963700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ad0100770 0x7f8ad019be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8ad6963700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad019c3d0 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8ad6963700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad019c510 con 0x7f8ad0100770 2023-07-14T19:04:16.071 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8ad6963700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad019c6a0 con 0x7f8ad00feb30 2023-07-14T19:04:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8acf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad00ffdc0 0x7f8ad0194d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8acf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad00ffdc0 0x7f8ad0194d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:35996/0 (socket says 172.21.15.78:35996) 2023-07-14T19:04:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.062+0000 7f8acf7fe700 1 -- 172.21.15.78:0/623781399 learned_addr learned my addr 172.21.15.78:0/623781399 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acffff700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad00feb30 0x7f8ad0194850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acf7fe700 1 -- 172.21.15.78:0/623781399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad00feb30 msgr2=0x7f8ad0194850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8ad4f00700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ad0100770 0x7f8ad019be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.072 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acf7fe700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad00feb30 0x7f8ad0194850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acf7fe700 1 -- 172.21.15.78:0/623781399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ad0100770 msgr2=0x7f8ad019be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acf7fe700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ad0100770 0x7f8ad019be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acf7fe700 1 -- 172.21.15.78:0/623781399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ac8009d00 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acf7fe700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad00ffdc0 0x7f8ad0194d90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400eab0 tx=0x7f8ac400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acd7fa700 1 -- 172.21.15.78:0/623781399 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4013070 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.073 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acd7fa700 1 -- 172.21.15.78:0/623781399 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f8ac4017410 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad00689b0 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8acd7fa700 1 -- 172.21.15.78:0/623781399 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4016610 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.063+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad0068ec0 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.064+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad010a160 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.065+0000 7f8acd7fa700 1 -- 172.21.15.78:0/623781399 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f8ac4017580 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.065+0000 7f8acd7fa700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ab80a1150 0x7f8ab80a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.066+0000 7f8acffff700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ab80a1150 0x7f8ab80a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.074 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.066+0000 7f8acd7fa700 1 -- 172.21.15.78:0/623781399 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f8ac40ce560 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.066+0000 7f8acffff700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ab80a1150 0x7f8ab80a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0005950 tx=0x7f8ac00058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:16.075 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.069+0000 7f8acd7fa700 1 -- 172.21.15.78:0/623781399 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f8ac4097940 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.262 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.259+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f8ad0105b20 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.263 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.259+0000 7f8acd7fa700 1 -- 172.21.15.78:0/623781399 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7f8ac409af60 con 0x7f8ad00ffdc0 2023-07-14T19:04:16.263 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:04:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ab80a1150 msgr2=0x7f8ab80a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ab80a1150 0x7f8ab80a3600 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f8ac0005950 tx=0x7f8ac00058e0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad00ffdc0 msgr2=0x7f8ad0194d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad00ffdc0 0x7f8ad0194d90 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400eab0 tx=0x7f8ac400ee70 comp rx=0 tx=0).stop 2023-07-14T19:04:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 shutdown_connections 2023-07-14T19:04:16.265 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f8ab80a1150 0x7f8ab80a3600 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f8ad0100770 0x7f8ad019be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad00feb30 0x7f8ad0194850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 --2- 172.21.15.78:0/623781399 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f8ad00ffdc0 0x7f8ad0194d90 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.261+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 >> 172.21.15.78:0/623781399 conn(0x7f8ad00fa150 msgr2=0x7f8ad00fc3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.262+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 shutdown_connections 2023-07-14T19:04:16.266 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.262+0000 7f8ad6963700 1 -- 172.21.15.78:0/623781399 wait complete. 2023-07-14T19:04:16.277 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:04:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 -- 172.21.15.78:0/4223954714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 msgr2=0x7fc5480ff1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 --2- 172.21.15.78:0/4223954714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 0x7fc5480ff1b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fc534009990 tx=0x7fc534009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 -- 172.21.15.78:0/4223954714 shutdown_connections 2023-07-14T19:04:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 --2- 172.21.15.78:0/4223954714 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc548100180 0x7fc548100590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 --2- 172.21.15.78:0/4223954714 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc5480ff6f0 0x7fc548106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 --2- 172.21.15.78:0/4223954714 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 0x7fc5480ff1b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 -- 172.21.15.78:0/4223954714 >> 172.21.15.78:0/4223954714 conn(0x7fc5480f98a0 msgr2=0x7fc5480fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 -- 172.21.15.78:0/4223954714 shutdown_connections 2023-07-14T19:04:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.394+0000 7fc54eeee700 1 -- 172.21.15.78:0/4223954714 wait complete. 2023-07-14T19:04:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54eeee700 1 Processor -- start 2023-07-14T19:04:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54eeee700 1 -- start start 2023-07-14T19:04:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54eeee700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 0x7fc5481957d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54eeee700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5480ff6f0 0x7fc548195d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54eeee700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc548100180 0x7fc54819cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54eeee700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54819d350 con 0x7fc5480fed40 2023-07-14T19:04:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54eeee700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54819d490 con 0x7fc5480ff6f0 2023-07-14T19:04:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54eeee700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc54819d620 con 0x7fc548100180 2023-07-14T19:04:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54cc8a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 0x7fc5481957d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54cc8a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 0x7fc5481957d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36008/0 (socket says 172.21.15.78:36008) 2023-07-14T19:04:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54cc8a700 1 -- 172.21.15.78:0/735262884 learned_addr learned my addr 172.21.15.78:0/735262884 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc547fff700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5480ff6f0 0x7fc548195d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54d48b700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc548100180 0x7fc54819cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54cc8a700 1 -- 172.21.15.78:0/735262884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc548100180 msgr2=0x7fc54819cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54cc8a700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc548100180 0x7fc54819cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54cc8a700 1 -- 172.21.15.78:0/735262884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5480ff6f0 msgr2=0x7fc548195d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54cc8a700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5480ff6f0 0x7fc548195d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.395+0000 7fc54cc8a700 1 -- 172.21.15.78:0/735262884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5340095f0 con 0x7fc5480fed40 2023-07-14T19:04:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.396+0000 7fc54cc8a700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 0x7fc5481957d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00e9a0 tx=0x7fc53c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.396+0000 7fc545ffb700 1 -- 172.21.15.78:0/735262884 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c009790 con 0x7fc5480fed40 2023-07-14T19:04:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.396+0000 7fc545ffb700 1 -- 172.21.15.78:0/735262884 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc53c00cbd0 con 0x7fc5480fed40 2023-07-14T19:04:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.396+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54819d900 con 0x7fc5480fed40 2023-07-14T19:04:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.396+0000 7fc545ffb700 1 -- 172.21.15.78:0/735262884 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc53c018680 con 0x7fc5480fed40 2023-07-14T19:04:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.396+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54819de10 con 0x7fc5480fed40 2023-07-14T19:04:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.397+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54810a180 con 0x7fc5480fed40 2023-07-14T19:04:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.398+0000 7fc545ffb700 1 -- 172.21.15.78:0/735262884 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc53c00cd40 con 0x7fc5480fed40 2023-07-14T19:04:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.398+0000 7fc545ffb700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc5380a11a0 0x7fc5380a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.398+0000 7fc545ffb700 1 -- 172.21.15.78:0/735262884 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc53c014070 con 0x7fc5480fed40 2023-07-14T19:04:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.399+0000 7fc547fff700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc5380a11a0 0x7fc5380a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.399+0000 7fc547fff700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc5380a11a0 0x7fc5380a3650 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc534006010 tx=0x7fc534005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:16.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.402+0000 7fc545ffb700 1 -- 172.21.15.78:0/735262884 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc53c096640 con 0x7fc5480fed40 2023-07-14T19:04:16.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:16 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1285872436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:16 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2915099773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:16 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3502735711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:16 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1285872436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:16 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2915099773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:16 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3502735711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.637 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.634+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc54810a340 con 0x7fc5480fed40 2023-07-14T19:04:16.639 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.636+0000 7fc545ffb700 1 -- 172.21.15.78:0/735262884 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10808 (secure 0 0 0) 0x7fc53c099c60 con 0x7fc5480fed40 2023-07-14T19:04:16.640 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:16.640 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:02:50.605732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24485":{"gid":24485,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:04:16.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc5380a11a0 msgr2=0x7fc5380a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc5380a11a0 0x7fc5380a3650 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc534006010 tx=0x7fc534005960 comp rx=0 tx=0).stop 2023-07-14T19:04:16.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 msgr2=0x7fc5481957d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 0x7fc5481957d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fc53c00e9a0 tx=0x7fc53c00ecb0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.642 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 shutdown_connections 2023-07-14T19:04:16.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc5380a11a0 0x7fc5380a3650 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc548100180 0x7fc54819cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc5480fed40 0x7fc5481957d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 --2- 172.21.15.78:0/735262884 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc5480ff6f0 0x7fc548195d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.638+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 >> 172.21.15.78:0/735262884 conn(0x7fc5480f98a0 msgr2=0x7fc5480fdcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:16.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.639+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 shutdown_connections 2023-07-14T19:04:16.643 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.639+0000 7fc54eeee700 1 -- 172.21.15.78:0/735262884 wait complete. 2023-07-14T19:04:16.644 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2023-07-14T19:04:16.655 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:04:16.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:16 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1285872436' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:16 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2915099773' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:16 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3502735711' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:16.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 -- 172.21.15.78:0/644063432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 msgr2=0x7f62c40fef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 --2- 172.21.15.78:0/644063432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 0x7f62c40fef40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f62c0009990 tx=0x7f62c0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 -- 172.21.15.78:0/644063432 shutdown_connections 2023-07-14T19:04:16.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 --2- 172.21.15.78:0/644063432 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62c41006b0 0x7f62c4106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 --2- 172.21.15.78:0/644063432 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 0x7f62c40fef40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 --2- 172.21.15.78:0/644063432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c40ffd20 0x7f62c4100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 -- 172.21.15.78:0/644063432 >> 172.21.15.78:0/644063432 conn(0x7f62c40fa110 msgr2=0x7f62c40fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:16.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 -- 172.21.15.78:0/644063432 shutdown_connections 2023-07-14T19:04:16.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.772+0000 7f62cc751700 1 -- 172.21.15.78:0/644063432 wait complete. 2023-07-14T19:04:16.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cc751700 1 Processor -- start 2023-07-14T19:04:16.778 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cc751700 1 -- start start 2023-07-14T19:04:16.779 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cc751700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 0x7f62c4102ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cc751700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62c40ffd20 0x7f62c4101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cc751700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41006b0 0x7f62c4101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cc751700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4101de0 con 0x7f62c40feb30 2023-07-14T19:04:16.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cc751700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c4101f50 con 0x7f62c40ffd20 2023-07-14T19:04:16.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cc751700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c419f8e0 con 0x7f62c41006b0 2023-07-14T19:04:16.780 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62ca4ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 0x7f62c4102ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62ca4ed700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 0x7f62c4102ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36022/0 (socket says 172.21.15.78:36022) 2023-07-14T19:04:16.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62ca4ed700 1 -- 172.21.15.78:0/3209082852 learned_addr learned my addr 172.21.15.78:0/3209082852 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:16.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62c9cec700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62c40ffd20 0x7f62c4101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62cacee700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41006b0 0x7f62c4101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62ca4ed700 1 -- 172.21.15.78:0/3209082852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41006b0 msgr2=0x7f62c4101640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.773+0000 7f62ca4ed700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41006b0 0x7f62c4101640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.781 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62ca4ed700 1 -- 172.21.15.78:0/3209082852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62c40ffd20 msgr2=0x7f62c4101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62ca4ed700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62c40ffd20 0x7f62c4101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62ca4ed700 1 -- 172.21.15.78:0/3209082852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c00095f0 con 0x7f62c40feb30 2023-07-14T19:04:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62c9cec700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62c40ffd20 0x7f62c4101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62cacee700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41006b0 0x7f62c4101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62ca4ed700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 0x7f62c4102ac0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f62c0000c00 tx=0x7f62c0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62bb7fe700 1 -- 172.21.15.78:0/3209082852 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c001d070 con 0x7f62c40feb30 2023-07-14T19:04:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62bb7fe700 1 -- 172.21.15.78:0/3209082852 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f62c00040b0 con 0x7f62c40feb30 2023-07-14T19:04:16.782 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c419fb60 con 0x7f62c40feb30 2023-07-14T19:04:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62bb7fe700 1 -- 172.21.15.78:0/3209082852 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62c0022cb0 con 0x7f62c40feb30 2023-07-14T19:04:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.774+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c41a0010 con 0x7f62c40feb30 2023-07-14T19:04:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.775+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c410a1b0 con 0x7f62c40feb30 2023-07-14T19:04:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.776+0000 7f62bb7fe700 1 -- 172.21.15.78:0/3209082852 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f62c0022470 con 0x7f62c40feb30 2023-07-14T19:04:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.776+0000 7f62bb7fe700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f62b00a11a0 0x7f62b00a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.776+0000 7f62c9cec700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f62b00a11a0 0x7f62b00a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:16.783 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.776+0000 7f62bb7fe700 1 -- 172.21.15.78:0/3209082852 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f62c00ce120 con 0x7f62c40feb30 2023-07-14T19:04:16.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.777+0000 7f62c9cec700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f62b00a11a0 0x7f62b00a3650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f62b4005950 tx=0x7f62b40058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:16.784 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.780+0000 7f62bb7fe700 1 -- 172.21.15.78:0/3209082852 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f62c00975f0 con 0x7f62c40feb30 2023-07-14T19:04:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.971+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62c41a05c0 con 0x7f62c40feb30 2023-07-14T19:04:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.973+0000 7f62bb7fe700 1 -- 172.21.15.78:0/3209082852 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24453 (secure 0 0 0) 0x7f62c009ac10 con 0x7f62c40feb30 2023-07-14T19:04:16.977 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:16.977 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:03:31.201384+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:04:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f62b00a11a0 msgr2=0x7f62b00a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f62b00a11a0 0x7f62b00a3650 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f62b4005950 tx=0x7f62b40058e0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.979 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 msgr2=0x7f62c4102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 0x7f62c4102ac0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f62c0000c00 tx=0x7f62c0009f90 comp rx=0 tx=0).stop 2023-07-14T19:04:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 shutdown_connections 2023-07-14T19:04:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f62b00a11a0 0x7f62b00a3650 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f62c41006b0 0x7f62c4101640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f62c40feb30 0x7f62c4102ac0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 --2- 172.21.15.78:0/3209082852 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f62c40ffd20 0x7f62c4101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:16.980 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 >> 172.21.15.78:0/3209082852 conn(0x7f62c40fa110 msgr2=0x7f62c4068870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:16.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 shutdown_connections 2023-07-14T19:04:16.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:16.976+0000 7f62cc751700 1 -- 172.21.15.78:0/3209082852 wait complete. 2023-07-14T19:04:16.992 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:02:28.269050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:02:29.547771+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T19:02:32.797605+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T19:04:16.993 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:04:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 -- 172.21.15.78:0/4276989240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 msgr2=0x7f4260106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 --2- 172.21.15.78:0/4276989240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 0x7f4260106f50 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f425800b0a0 tx=0x7f425800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 -- 172.21.15.78:0/4276989240 shutdown_connections 2023-07-14T19:04:17.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 --2- 172.21.15.78:0/4276989240 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 0x7f4260106f50 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 --2- 172.21.15.78:0/4276989240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42600feb10 0x7f42600fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 --2- 172.21.15.78:0/4276989240 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42600ffd00 0x7f4260100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 -- 172.21.15.78:0/4276989240 >> 172.21.15.78:0/4276989240 conn(0x7f42600fa130 msgr2=0x7f42600fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:17.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 -- 172.21.15.78:0/4276989240 shutdown_connections 2023-07-14T19:04:17.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 -- 172.21.15.78:0/4276989240 wait complete. 2023-07-14T19:04:17.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.114+0000 7f4265408700 1 Processor -- start 2023-07-14T19:04:17.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4265408700 1 -- start start 2023-07-14T19:04:17.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4265408700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42600feb10 0x7f4260068310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4265408700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42600ffd00 0x7f426019a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4265408700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 0x7f426019cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4265408700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426019d4d0 con 0x7f4260100690 2023-07-14T19:04:17.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4265408700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426019d640 con 0x7f42600ffd00 2023-07-14T19:04:17.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4265408700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f426019d800 con 0x7f42600feb10 2023-07-14T19:04:17.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 0x7f426019cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 0x7f426019cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36038/0 (socket says 172.21.15.78:36038) 2023-07-14T19:04:17.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 -- 172.21.15.78:0/3634549514 learned_addr learned my addr 172.21.15.78:0/3634549514 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:17.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425e7fc700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42600ffd00 0x7f426019a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425effd700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42600feb10 0x7f4260068310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 -- 172.21.15.78:0/3634549514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42600feb10 msgr2=0x7f4260068310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42600feb10 0x7f4260068310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 -- 172.21.15.78:0/3634549514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42600ffd00 msgr2=0x7f426019a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42600ffd00 0x7f426019a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 -- 172.21.15.78:0/3634549514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4258009d00 con 0x7f4260100690 2023-07-14T19:04:17.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f425f7fe700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 0x7f426019cd30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f425800b830 tx=0x7f4258004ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:17.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4247fff700 1 -- 172.21.15.78:0/3634549514 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425800e040 con 0x7f4260100690 2023-07-14T19:04:17.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4247fff700 1 -- 172.21.15.78:0/3634549514 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4258009440 con 0x7f4260100690 2023-07-14T19:04:17.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f426019da80 con 0x7f4260100690 2023-07-14T19:04:17.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.115+0000 7f4247fff700 1 -- 172.21.15.78:0/3634549514 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f425801db60 con 0x7f4260100690 2023-07-14T19:04:17.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.116+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f426019dea0 con 0x7f4260100690 2023-07-14T19:04:17.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.116+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4260106090 con 0x7f4260100690 2023-07-14T19:04:17.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.118+0000 7f4247fff700 1 -- 172.21.15.78:0/3634549514 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f4258005450 con 0x7f4260100690 2023-07-14T19:04:17.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.119+0000 7f4247fff700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42480a1150 0x7f42480a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.119+0000 7f4247fff700 1 -- 172.21.15.78:0/3634549514 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f42580cebb0 con 0x7f4260100690 2023-07-14T19:04:17.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.119+0000 7f425effd700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42480a1150 0x7f42480a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.119+0000 7f425effd700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42480a1150 0x7f42480a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f42600ffb60 tx=0x7f425400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:17.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.122+0000 7f4247fff700 1 -- 172.21.15.78:0/3634549514 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4258098020 con 0x7f4260100690 2023-07-14T19:04:17.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:17 smithi078 ceph-mon[127425]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2023-07-14T19:04:17.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:17 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/623781399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:17.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:17 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/735262884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:17.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:17 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3209082852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.314+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f426010a450 con 0x7f4260100690 2023-07-14T19:04:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.315+0000 7f4247fff700 1 -- 172.21.15.78:0/3634549514 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4258005700 con 0x7f4260100690 2023-07-14T19:04:17.318 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:04:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42480a1150 msgr2=0x7f42480a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42480a1150 0x7f42480a3600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f42600ffb60 tx=0x7f425400a380 comp rx=0 tx=0).stop 2023-07-14T19:04:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 msgr2=0x7f426019cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 0x7f426019cd30 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f425800b830 tx=0x7f4258004ce0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 shutdown_connections 2023-07-14T19:04:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f42480a1150 0x7f42480a3600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4260100690 0x7f426019cd30 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f42600feb10 0x7f4260068310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 --2- 172.21.15.78:0/3634549514 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f42600ffd00 0x7f426019a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 >> 172.21.15.78:0/3634549514 conn(0x7f42600fa130 msgr2=0x7f42601031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 shutdown_connections 2023-07-14T19:04:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.317+0000 7f4265408700 1 -- 172.21.15.78:0/3634549514 wait complete. 2023-07-14T19:04:17.333 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:04:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 -- 172.21.15.78:0/338406637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c105970 msgr2=0x7fed0c107d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 --2- 172.21.15.78:0/338406637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c105970 0x7fed0c107d70 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fed0400b0a0 tx=0x7fed0400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 -- 172.21.15.78:0/338406637 shutdown_connections 2023-07-14T19:04:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 --2- 172.21.15.78:0/338406637 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed0c100730 0x7fed0c102b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.458 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 --2- 172.21.15.78:0/338406637 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c105970 0x7fed0c107d70 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 --2- 172.21.15.78:0/338406637 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c103050 0x7fed0c105430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 -- 172.21.15.78:0/338406637 >> 172.21.15.78:0/338406637 conn(0x7fed0c0fa150 msgr2=0x7fed0c0fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:17.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 -- 172.21.15.78:0/338406637 shutdown_connections 2023-07-14T19:04:17.459 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.454+0000 7fed10ea3700 1 -- 172.21.15.78:0/338406637 wait complete. 2023-07-14T19:04:17.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed10ea3700 1 Processor -- start 2023-07-14T19:04:17.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed10ea3700 1 -- start start 2023-07-14T19:04:17.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed10ea3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c100730 0x7fed0c195990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed10ea3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c103050 0x7fed0c195ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed10ea3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed0c105970 0x7fed0c19cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.461 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed10ea3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c19d510 con 0x7fed0c100730 2023-07-14T19:04:17.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed10ea3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c19d650 con 0x7fed0c105970 2023-07-14T19:04:17.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed10ea3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c19d7e0 con 0x7fed0c103050 2023-07-14T19:04:17.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c100730 0x7fed0c195990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c100730 0x7fed0c195990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36048/0 (socket says 172.21.15.78:36048) 2023-07-14T19:04:17.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 -- 172.21.15.78:0/690475337 learned_addr learned my addr 172.21.15.78:0/690475337 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:17.462 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed09d9b700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c103050 0x7fed0c195ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0ad9d700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed0c105970 0x7fed0c19cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 -- 172.21.15.78:0/690475337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c103050 msgr2=0x7fed0c195ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c103050 0x7fed0c195ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 -- 172.21.15.78:0/690475337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed0c105970 msgr2=0x7fed0c19cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.463 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed0c105970 0x7fed0c19cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 -- 172.21.15.78:0/690475337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed04009d00 con 0x7fed0c100730 2023-07-14T19:04:17.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.455+0000 7fed0a59c700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c100730 0x7fed0c195990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fed0000d980 tx=0x7fed0000dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:17.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.456+0000 7fecfb7fe700 1 -- 172.21.15.78:0/690475337 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0000bb60 con 0x7fed0c100730 2023-07-14T19:04:17.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.456+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c19dac0 con 0x7fed0c100730 2023-07-14T19:04:17.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.456+0000 7fecfb7fe700 1 -- 172.21.15.78:0/690475337 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fed00007640 con 0x7fed0c100730 2023-07-14T19:04:17.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.456+0000 7fecfb7fe700 1 -- 172.21.15.78:0/690475337 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00004d60 con 0x7fed0c100730 2023-07-14T19:04:17.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.456+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c19dfd0 con 0x7fed0c100730 2023-07-14T19:04:17.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.457+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c0fbcc0 con 0x7fed0c100730 2023-07-14T19:04:17.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.458+0000 7fecfb7fe700 1 -- 172.21.15.78:0/690475337 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fed00004020 con 0x7fed0c100730 2023-07-14T19:04:17.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.458+0000 7fecfb7fe700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a1150 0x7fecf40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.458+0000 7fed09d9b700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a1150 0x7fecf40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.458+0000 7fecfb7fe700 1 -- 172.21.15.78:0/690475337 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fed000ccb60 con 0x7fed0c100730 2023-07-14T19:04:17.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.458+0000 7fed09d9b700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a1150 0x7fecf40a3600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fecfc005fd0 tx=0x7fecfc005d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:17.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.462+0000 7fecfb7fe700 1 -- 172.21.15.78:0/690475337 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fed00095fa0 con 0x7fed0c100730 2023-07-14T19:04:17.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:17 smithi102 ceph-mon[129280]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2023-07-14T19:04:17.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:17 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/623781399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:17.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:17 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/735262884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:17.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:17 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3209082852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:17.653 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.650+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fed0c02cd70 con 0x7fed0c100730 2023-07-14T19:04:17.654 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.651+0000 7fecfb7fe700 1 -- 172.21.15.78:0/690475337 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v87) v1 ==== 104+0+11 (secure 0 0 0) 0x7fed000160a0 con 0x7fed0c100730 2023-07-14T19:04:17.654 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:04:17.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a1150 msgr2=0x7fecf40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a1150 0x7fecf40a3600 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fecfc005fd0 tx=0x7fecfc005d90 comp rx=0 tx=0).stop 2023-07-14T19:04:17.656 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c100730 msgr2=0x7fed0c195990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c100730 0x7fed0c195990 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fed0000d980 tx=0x7fed0000dd40 comp rx=0 tx=0).stop 2023-07-14T19:04:17.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 shutdown_connections 2023-07-14T19:04:17.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fecf40a1150 0x7fecf40a3600 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fed0c105970 0x7fed0c19cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fed0c100730 0x7fed0c195990 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 --2- 172.21.15.78:0/690475337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fed0c103050 0x7fed0c195ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 >> 172.21.15.78:0/690475337 conn(0x7fed0c0fa150 msgr2=0x7fed0c101390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:17.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 shutdown_connections 2023-07-14T19:04:17.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.653+0000 7fed10ea3700 1 -- 172.21.15.78:0/690475337 wait complete. 2023-07-14T19:04:17.668 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:04:17.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:17 smithi088 ceph-mon[128973]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 85 B/s wr, 5 op/s 2023-07-14T19:04:17.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:17 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/623781399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:17.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:17 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/735262884' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:17.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:17 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3209082852' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:17.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 -- 172.21.15.78:0/3481114687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93981017a0 msgr2=0x7f9398108060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 --2- 172.21.15.78:0/3481114687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93981017a0 0x7f9398108060 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f939000b0a0 tx=0x7f939000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 -- 172.21.15.78:0/3481114687 shutdown_connections 2023-07-14T19:04:17.793 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 --2- 172.21.15.78:0/3481114687 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93980ffc20 0x7f9398100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 --2- 172.21.15.78:0/3481114687 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93981017a0 0x7f9398108060 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 --2- 172.21.15.78:0/3481114687 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398100e10 0x7f9398101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 -- 172.21.15.78:0/3481114687 >> 172.21.15.78:0/3481114687 conn(0x7f93980fb260 msgr2=0x7f93980fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:17.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 -- 172.21.15.78:0/3481114687 shutdown_connections 2023-07-14T19:04:17.794 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.789+0000 7f939d109700 1 -- 172.21.15.78:0/3481114687 wait complete. 2023-07-14T19:04:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f939d109700 1 Processor -- start 2023-07-14T19:04:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f939d109700 1 -- start start 2023-07-14T19:04:17.796 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f939d109700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980ffc20 0x7f93980743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f939d109700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398100e10 0x7f9398072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f939d109700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93981017a0 0x7f9398072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f939d109700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9398073710 con 0x7f93980ffc20 2023-07-14T19:04:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f939d109700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9398073880 con 0x7f93981017a0 2023-07-14T19:04:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f939d109700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f939819f9b0 con 0x7f9398100e10 2023-07-14T19:04:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f9396d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980ffc20 0x7f93980743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f9396d9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980ffc20 0x7f93980743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36062/0 (socket says 172.21.15.78:36062) 2023-07-14T19:04:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.790+0000 7f9396d9d700 1 -- 172.21.15.78:0/337770418 learned_addr learned my addr 172.21.15.78:0/337770418 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f939759e700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93981017a0 0x7f9398072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f939659c700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398100e10 0x7f9398072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f9396d9d700 1 -- 172.21.15.78:0/337770418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398100e10 msgr2=0x7f9398072a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f9396d9d700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398100e10 0x7f9398072a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f9396d9d700 1 -- 172.21.15.78:0/337770418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93981017a0 msgr2=0x7f9398072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f9396d9d700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93981017a0 0x7f9398072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f9396d9d700 1 -- 172.21.15.78:0/337770418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9390009d00 con 0x7f93980ffc20 2023-07-14T19:04:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f939759e700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93981017a0 0x7f9398072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f939659c700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398100e10 0x7f9398072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f9396d9d700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980ffc20 0x7f93980743f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f938c00e970 tx=0x7f938c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f937ffff700 1 -- 172.21.15.78:0/337770418 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c009790 con 0x7f93980ffc20 2023-07-14T19:04:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f937ffff700 1 -- 172.21.15.78:0/337770418 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f938c0044b0 con 0x7f93980ffc20 2023-07-14T19:04:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f939819fc90 con 0x7f93980ffc20 2023-07-14T19:04:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f937ffff700 1 -- 172.21.15.78:0/337770418 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f938c003dd0 con 0x7f93980ffc20 2023-07-14T19:04:17.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.791+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93981a01a0 con 0x7f93980ffc20 2023-07-14T19:04:17.800 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.792+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f939810b2b0 con 0x7f93980ffc20 2023-07-14T19:04:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.793+0000 7f937ffff700 1 -- 172.21.15.78:0/337770418 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f938c004000 con 0x7f93980ffc20 2023-07-14T19:04:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.794+0000 7f937ffff700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93800a11d0 0x7f93800a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.794+0000 7f939659c700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93800a11d0 0x7f93800a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.794+0000 7f937ffff700 1 -- 172.21.15.78:0/337770418 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f938c014070 con 0x7f93980ffc20 2023-07-14T19:04:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.794+0000 7f939659c700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93800a11d0 0x7f93800a3680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9388009cb0 tx=0x7f9388009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:17.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.797+0000 7f937ffff700 1 -- 172.21.15.78:0/337770418 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f938c0963f0 con 0x7f93980ffc20 2023-07-14T19:04:17.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.986+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9398106ad0 con 0x7f93980ffc20 2023-07-14T19:04:17.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.986+0000 7f937ffff700 1 -- 172.21.15.78:0/337770418 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v87) v1 ==== 100+0+11 (secure 0 0 0) 0x7f938c099a10 con 0x7f93980ffc20 2023-07-14T19:04:17.990 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-07-14T19:04:17.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93800a11d0 msgr2=0x7f93800a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93800a11d0 0x7f93800a3680 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9388009cb0 tx=0x7f9388009260 comp rx=0 tx=0).stop 2023-07-14T19:04:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980ffc20 msgr2=0x7f93980743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980ffc20 0x7f93980743f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f938c00e970 tx=0x7f938c00ed30 comp rx=0 tx=0).stop 2023-07-14T19:04:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 shutdown_connections 2023-07-14T19:04:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f93800a11d0 0x7f93800a3680 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f93981017a0 0x7f9398072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f93980ffc20 0x7f93980743f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 --2- 172.21.15.78:0/337770418 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9398100e10 0x7f9398072a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 >> 172.21.15.78:0/337770418 conn(0x7f93980fb260 msgr2=0x7f93980fd440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 shutdown_connections 2023-07-14T19:04:17.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:17.988+0000 7f939d109700 1 -- 172.21.15.78:0/337770418 wait complete. 2023-07-14T19:04:18.005 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T19:04:18.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1084090471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c8100690 msgr2=0x7fc0c8106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1084090471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c8100690 0x7fc0c8106f50 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000b0a0 tx=0x7fc0c000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1084090471 shutdown_connections 2023-07-14T19:04:18.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1084090471 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c8100690 0x7fc0c8106f50 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1084090471 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c80feb10 0x7fc0c80fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1084090471 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0c80ffd00 0x7fc0c8100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1084090471 >> 172.21.15.78:0/1084090471 conn(0x7fc0c80fa130 msgr2=0x7fc0c80fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:18.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1084090471 shutdown_connections 2023-07-14T19:04:18.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.125+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1084090471 wait complete. 2023-07-14T19:04:18.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cf6c3700 1 Processor -- start 2023-07-14T19:04:18.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cf6c3700 1 -- start start 2023-07-14T19:04:18.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cf6c3700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0c80feb10 0x7fc0c8195980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cf6c3700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c80ffd00 0x7fc0c8195ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cf6c3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c8100690 0x7fc0c819cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cf6c3700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c819d500 con 0x7fc0c80ffd00 2023-07-14T19:04:18.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cf6c3700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c819d640 con 0x7fc0c80feb10 2023-07-14T19:04:18.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cf6c3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c819d7d0 con 0x7fc0c8100690 2023-07-14T19:04:18.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0ccc5e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c80ffd00 0x7fc0c8195ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0ccc5e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c80ffd00 0x7fc0c8195ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36076/0 (socket says 172.21.15.78:36076) 2023-07-14T19:04:18.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0ccc5e700 1 -- 172.21.15.78:0/1874159743 learned_addr learned my addr 172.21.15.78:0/1874159743 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:18.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cd45f700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0c80feb10 0x7fc0c8195980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0cdc60700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c8100690 0x7fc0c819cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0ccc5e700 1 -- 172.21.15.78:0/1874159743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c8100690 msgr2=0x7fc0c819cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0ccc5e700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c8100690 0x7fc0c819cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0ccc5e700 1 -- 172.21.15.78:0/1874159743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0c80feb10 msgr2=0x7fc0c8195980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0ccc5e700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0c80feb10 0x7fc0c8195980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.126+0000 7fc0ccc5e700 1 -- 172.21.15.78:0/1874159743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c0009d00 con 0x7fc0c80ffd00 2023-07-14T19:04:18.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.127+0000 7fc0ccc5e700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c80ffd00 0x7fc0c8195ec0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400c920 tx=0x7fc0c400cc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:18.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.127+0000 7fc0ba7fc700 1 -- 172.21.15.78:0/1874159743 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4007b00 con 0x7fc0c80ffd00 2023-07-14T19:04:18.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.127+0000 7fc0ba7fc700 1 -- 172.21.15.78:0/1874159743 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc0c400ce70 con 0x7fc0c80ffd00 2023-07-14T19:04:18.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.127+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c819dab0 con 0x7fc0c80ffd00 2023-07-14T19:04:18.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.127+0000 7fc0ba7fc700 1 -- 172.21.15.78:0/1874159743 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4005440 con 0x7fc0c80ffd00 2023-07-14T19:04:18.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.127+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c819e010 con 0x7fc0c80ffd00 2023-07-14T19:04:18.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.128+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c810a1a0 con 0x7fc0c80ffd00 2023-07-14T19:04:18.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.129+0000 7fc0ba7fc700 1 -- 172.21.15.78:0/1874159743 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc0c40055a0 con 0x7fc0c80ffd00 2023-07-14T19:04:18.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.129+0000 7fc0ba7fc700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc0b40a11d0 0x7fc0b40a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.129+0000 7fc0cd45f700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc0b40a11d0 0x7fc0b40a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.129+0000 7fc0ba7fc700 1 -- 172.21.15.78:0/1874159743 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc0c40cc770 con 0x7fc0c80ffd00 2023-07-14T19:04:18.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.130+0000 7fc0cd45f700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc0b40a11d0 0x7fc0b40a3680 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80ffb60 tx=0x7fc0bc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:18.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.133+0000 7fc0ba7fc700 1 -- 172.21.15.78:0/1874159743 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc0c4095bb0 con 0x7fc0c80ffd00 2023-07-14T19:04:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:18 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3634549514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:18 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/690475337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:18 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/337770418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.324+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc0c819e620 con 0x7fc0c80ffd00 2023-07-14T19:04:18.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.325+0000 7fc0ba7fc700 1 -- 172.21.15.78:0/1874159743 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v87) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc0c40991d0 con 0x7fc0c80ffd00 2023-07-14T19:04:18.329 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:04:18.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc0b40a11d0 msgr2=0x7fc0b40a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc0b40a11d0 0x7fc0b40a3680 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fc0c80ffb60 tx=0x7fc0bc00b410 comp rx=0 tx=0).stop 2023-07-14T19:04:18.330 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c80ffd00 msgr2=0x7fc0c8195ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c80ffd00 0x7fc0c8195ec0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400c920 tx=0x7fc0c400cc30 comp rx=0 tx=0).stop 2023-07-14T19:04:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 shutdown_connections 2023-07-14T19:04:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc0b40a11d0 0x7fc0b40a3680 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc0c80feb10 0x7fc0c8195980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0c8100690 0x7fc0c819cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 --2- 172.21.15.78:0/1874159743 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc0c80ffd00 0x7fc0c8195ec0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 >> 172.21.15.78:0/1874159743 conn(0x7fc0c80fa130 msgr2=0x7fc0c8067860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:18.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 shutdown_connections 2023-07-14T19:04:18.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.327+0000 7fc0cf6c3700 1 -- 172.21.15.78:0/1874159743 wait complete. 2023-07-14T19:04:18.342 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:04:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 -- 172.21.15.78:0/1558294596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 msgr2=0x7fe18c106f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1558294596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 0x7fe18c106f70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe18400b0a0 tx=0x7fe18400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 -- 172.21.15.78:0/1558294596 shutdown_connections 2023-07-14T19:04:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1558294596 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe18c0feb30 0x7fe18c0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1558294596 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 0x7fe18c106f70 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1558294596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18c0ffd20 0x7fe18c100170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 -- 172.21.15.78:0/1558294596 >> 172.21.15.78:0/1558294596 conn(0x7fe18c0fa150 msgr2=0x7fe18c0fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 -- 172.21.15.78:0/1558294596 shutdown_connections 2023-07-14T19:04:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.460+0000 7fe190dd8700 1 -- 172.21.15.78:0/1558294596 wait complete. 2023-07-14T19:04:18.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe190dd8700 1 Processor -- start 2023-07-14T19:04:18.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe190dd8700 1 -- start start 2023-07-14T19:04:18.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe190dd8700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe18c0feb30 0x7fe18c102ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe190dd8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18c0ffd20 0x7fe18c101100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe190dd8700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 0x7fe18c101640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe190dd8700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c101de0 con 0x7fe18c1006b0 2023-07-14T19:04:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe190dd8700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c101f50 con 0x7fe18c0feb30 2023-07-14T19:04:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe190dd8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe18c19f900 con 0x7fe18c0ffd20 2023-07-14T19:04:18.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe18ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 0x7fe18c101640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe18ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 0x7fe18c101640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36092/0 (socket says 172.21.15.78:36092) 2023-07-14T19:04:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe18ad9d700 1 -- 172.21.15.78:0/1566172531 learned_addr learned my addr 172.21.15.78:0/1566172531 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.461+0000 7fe189d9b700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18c0ffd20 0x7fe18c101100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe18ad9d700 1 -- 172.21.15.78:0/1566172531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18c0ffd20 msgr2=0x7fe18c101100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe18a59c700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe18c0feb30 0x7fe18c102ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe18ad9d700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18c0ffd20 0x7fe18c101100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe18ad9d700 1 -- 172.21.15.78:0/1566172531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe18c0feb30 msgr2=0x7fe18c102ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe18ad9d700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe18c0feb30 0x7fe18c102ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe18ad9d700 1 -- 172.21.15.78:0/1566172531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe184009d00 con 0x7fe18c1006b0 2023-07-14T19:04:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe189d9b700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18c0ffd20 0x7fe18c101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe18ad9d700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 0x7fe18c101640 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe1840062a0 tx=0x7fe184004d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe1837fe700 1 -- 172.21.15.78:0/1566172531 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18400e040 con 0x7fe18c1006b0 2023-07-14T19:04:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe1837fe700 1 -- 172.21.15.78:0/1566172531 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe18401bcf0 con 0x7fe18c1006b0 2023-07-14T19:04:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe18c19fb80 con 0x7fe18c1006b0 2023-07-14T19:04:18.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe1837fe700 1 -- 172.21.15.78:0/1566172531 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1840058e0 con 0x7fe18c1006b0 2023-07-14T19:04:18.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.462+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe18c1a0050 con 0x7fe18c1006b0 2023-07-14T19:04:18.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.463+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe18c10a1c0 con 0x7fe18c1006b0 2023-07-14T19:04:18.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.464+0000 7fe1837fe700 1 -- 172.21.15.78:0/1566172531 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fe184005040 con 0x7fe18c1006b0 2023-07-14T19:04:18.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.464+0000 7fe1837fe700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe1780a1220 0x7fe1780a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.465+0000 7fe18a59c700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe1780a1220 0x7fe1780a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.465+0000 7fe1837fe700 1 -- 172.21.15.78:0/1566172531 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fe1840cf200 con 0x7fe18c1006b0 2023-07-14T19:04:18.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.465+0000 7fe18a59c700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe1780a1220 0x7fe1780a36d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0ffb80 tx=0x7fe17c005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:18.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.468+0000 7fe1837fe700 1 -- 172.21.15.78:0/1566172531 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe1840986d0 con 0x7fe18c1006b0 2023-07-14T19:04:18.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:18 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3634549514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:18 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/690475337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:18 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/337770418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.701+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe18c1a0600 con 0x7fe18c1006b0 2023-07-14T19:04:18.705 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.702+0000 7fe1837fe700 1 -- 172.21.15.78:0/1566172531 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+10808 (secure 0 0 0) 0x7fe18409bcf0 con 0x7fe18c1006b0 2023-07-14T19:04:18.706 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:18.706 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:02:50.605732+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24485":{"gid":24485,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:04:18.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe1780a1220 msgr2=0x7fe1780a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe1780a1220 0x7fe1780a36d0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0ffb80 tx=0x7fe17c005cf0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 msgr2=0x7fe18c101640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 0x7fe18c101640 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fe1840062a0 tx=0x7fe184004d90 comp rx=0 tx=0).stop 2023-07-14T19:04:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 shutdown_connections 2023-07-14T19:04:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe1780a1220 0x7fe1780a36d0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe18c0feb30 0x7fe18c102ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe18c1006b0 0x7fe18c101640 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 --2- 172.21.15.78:0/1566172531 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe18c0ffd20 0x7fe18c101100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 >> 172.21.15.78:0/1566172531 conn(0x7fe18c0fa150 msgr2=0x7fe18c068a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:18.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 shutdown_connections 2023-07-14T19:04:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.704+0000 7fe190dd8700 1 -- 172.21.15.78:0/1566172531 wait complete. 2023-07-14T19:04:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 23 2023-07-14T19:04:18.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:18 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3634549514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:18 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/690475337' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:18 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/337770418' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:18.720 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:04:18.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 -- 172.21.15.78:0/2119927697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80ffcf0 msgr2=0x7fd0f8100140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/2119927697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80ffcf0 0x7fd0f8100140 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fd0e4009990 tx=0x7fd0e4009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 -- 172.21.15.78:0/2119927697 shutdown_connections 2023-07-14T19:04:18.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/2119927697 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f80feb00 0x7fd0f80fef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/2119927697 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8100680 0x7fd0f8106f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.842 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/2119927697 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80ffcf0 0x7fd0f8100140 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 -- 172.21.15.78:0/2119927697 >> 172.21.15.78:0/2119927697 conn(0x7fd0f80fa100 msgr2=0x7fd0f80fc550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:18.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 -- 172.21.15.78:0/2119927697 shutdown_connections 2023-07-14T19:04:18.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 -- 172.21.15.78:0/2119927697 wait complete. 2023-07-14T19:04:18.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.838+0000 7fd0fedfa700 1 Processor -- start 2023-07-14T19:04:18.843 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fedfa700 1 -- start start 2023-07-14T19:04:18.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fedfa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80feb00 0x7fd0f8068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.845 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fedfa700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f80ffcf0 0x7fd0f819a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fedfa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8100680 0x7fd0f819cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fedfa700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f819d320 con 0x7fd0f80feb00 2023-07-14T19:04:18.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fedfa700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f819d490 con 0x7fd0f80ffcf0 2023-07-14T19:04:18.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fedfa700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0f819d650 con 0x7fd0f8100680 2023-07-14T19:04:18.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80feb00 0x7fd0f8068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.846 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80feb00 0x7fd0f8068330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36102/0 (socket says 172.21.15.78:36102) 2023-07-14T19:04:18.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 -- 172.21.15.78:0/4173388079 learned_addr learned my addr 172.21.15.78:0/4173388079 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:18.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0f7fff700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f80ffcf0 0x7fd0f819a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 -- 172.21.15.78:0/4173388079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8100680 msgr2=0x7fd0f819cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fd397700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8100680 0x7fd0f819cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8100680 0x7fd0f819cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 -- 172.21.15.78:0/4173388079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f80ffcf0 msgr2=0x7fd0f819a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:18.847 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f80ffcf0 0x7fd0f819a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:18.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 -- 172.21.15.78:0/4173388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0e40095f0 con 0x7fd0f80feb00 2023-07-14T19:04:18.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0f7fff700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f80ffcf0 0x7fd0f819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:18.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fd397700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8100680 0x7fd0f819cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:18.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fcb96700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80feb00 0x7fd0f8068330 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00d980 tx=0x7fd0ec00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:18.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0f5ffb700 1 -- 172.21.15.78:0/4173388079 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec00de60 con 0x7fd0f80feb00 2023-07-14T19:04:18.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0f819d930 con 0x7fd0f80feb00 2023-07-14T19:04:18.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0f5ffb700 1 -- 172.21.15.78:0/4173388079 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd0ec0044b0 con 0x7fd0f80feb00 2023-07-14T19:04:18.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.839+0000 7fd0f5ffb700 1 -- 172.21.15.78:0/4173388079 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec003b40 con 0x7fd0f80feb00 2023-07-14T19:04:18.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.840+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0f819de40 con 0x7fd0f80feb00 2023-07-14T19:04:18.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.841+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0f810a1d0 con 0x7fd0f80feb00 2023-07-14T19:04:18.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.841+0000 7fd0f5ffb700 1 -- 172.21.15.78:0/4173388079 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fd0ec003ca0 con 0x7fd0f80feb00 2023-07-14T19:04:18.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.842+0000 7fd0f5ffb700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd0e80a11a0 0x7fd0e80a3650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:18.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.842+0000 7fd0f7fff700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd0e80a11a0 0x7fd0e80a3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:18.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.842+0000 7fd0f5ffb700 1 -- 172.21.15.78:0/4173388079 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fd0ec0cc950 con 0x7fd0f80feb00 2023-07-14T19:04:18.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.842+0000 7fd0f7fff700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd0e80a11a0 0x7fd0e80a3650 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400b5c0 tx=0x7fd0e4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:18.850 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:18.846+0000 7fd0f5ffb700 1 -- 172.21.15.78:0/4173388079 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd0ec095d90 con 0x7fd0f80feb00 2023-07-14T19:04:19.039 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.036+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0f819e450 con 0x7fd0f80feb00 2023-07-14T19:04:19.040 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.037+0000 7fd0f5ffb700 1 -- 172.21.15.78:0/4173388079 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+24453 (secure 0 0 0) 0x7fd0ec009940 con 0x7fd0f80feb00 2023-07-14T19:04:19.041 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:19.041 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":87,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:03:31.201384+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:04:19.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd0e80a11a0 msgr2=0x7fd0e80a3650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd0e80a11a0 0x7fd0e80a3650 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd0e400b5c0 tx=0x7fd0e4005fd0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.043 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80feb00 msgr2=0x7fd0f8068330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80feb00 0x7fd0f8068330 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd0ec00d980 tx=0x7fd0ec00dd40 comp rx=0 tx=0).stop 2023-07-14T19:04:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 shutdown_connections 2023-07-14T19:04:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd0e80a11a0 0x7fd0e80a3650 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd0f8100680 0x7fd0f819cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.044 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd0f80feb00 0x7fd0f8068330 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 --2- 172.21.15.78:0/4173388079 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd0f80ffcf0 0x7fd0f819a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 >> 172.21.15.78:0/4173388079 conn(0x7fd0f80fa100 msgr2=0x7fd0f80fc320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 shutdown_connections 2023-07-14T19:04:19.045 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.039+0000 7fd0fedfa700 1 -- 172.21.15.78:0/4173388079 wait complete. 2023-07-14T19:04:19.055 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2023-07-14T19:02:30.340765+0000', 'modified': '2023-07-14T19:02:50.605732+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24503, 'mds_1': 24487, 'mds_2': 24485}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24457': {'gid': 24457, 'name': 'f', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.102:6835/75062394', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6834', 'nonce': 75062394}, {'type': 'v1', 'addr': '172.21.15.102:6835', 'nonce': 75062394}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24464': {'gid': 24464, 'name': 'e', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.88:6835/3667124043', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6834', 'nonce': 3667124043}, {'type': 'v1', 'addr': '172.21.15.88:6835', 'nonce': 3667124043}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24469': {'gid': 24469, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.102:6837/3933565644', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6836', 'nonce': 3933565644}, {'type': 'v1', 'addr': '172.21.15.102:6837', 'nonce': 3933565644}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24485': {'gid': 24485, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.88:6837/1950772288', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6836', 'nonce': 1950772288}, {'type': 'v1', 'addr': '172.21.15.88:6837', 'nonce': 1950772288}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24487': {'gid': 24487, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.102:6839/984471622', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6838', 'nonce': 984471622}, {'type': 'v1', 'addr': '172.21.15.102:6839', 'nonce': 984471622}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}, 'gid_24503': {'gid': 24503, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.88:6839/1379790346', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.88:6838', 'nonce': 1379790346}, {'type': 'v1', 'addr': '172.21.15.88:6839', 'nonce': 1379790346}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-07-14T19:04:19.056 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2023-07-14T19:04:19.056 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2023-07-14T19:04:19.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.174+0000 7f5090975700 1 -- 172.21.15.78:0/2834504644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 msgr2=0x7f508c0ff210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.174+0000 7f5090975700 1 --2- 172.21.15.78:0/2834504644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 0x7f508c0ff210 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f5080009990 tx=0x7f5080009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.178 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 -- 172.21.15.78:0/2834504644 shutdown_connections 2023-07-14T19:04:19.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 --2- 172.21.15.78:0/2834504644 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f508c0ff750 0x7f508c107000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 --2- 172.21.15.78:0/2834504644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f508c100150 0x7f508c100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 --2- 172.21.15.78:0/2834504644 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 0x7f508c0ff210 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 -- 172.21.15.78:0/2834504644 >> 172.21.15.78:0/2834504644 conn(0x7f508c0f98e0 msgr2=0x7f508c0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 -- 172.21.15.78:0/2834504644 shutdown_connections 2023-07-14T19:04:19.179 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 -- 172.21.15.78:0/2834504644 wait complete. 2023-07-14T19:04:19.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 Processor -- start 2023-07-14T19:04:19.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 -- start start 2023-07-14T19:04:19.181 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 0x7f508c1958a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f508c0ff750 0x7f508c195de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.175+0000 7f5090975700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f508c100150 0x7f508c19ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f5090975700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c19d420 con 0x7f508c0feda0 2023-07-14T19:04:19.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f5090975700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c19d560 con 0x7f508c0ff750 2023-07-14T19:04:19.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f5090975700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f508c19d6f0 con 0x7f508c100150 2023-07-14T19:04:19.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 0x7f508c1958a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.182 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 0x7f508c1958a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36116/0 (socket says 172.21.15.78:36116) 2023-07-14T19:04:19.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 -- 172.21.15.78:0/2656189742 learned_addr learned my addr 172.21.15.78:0/2656189742 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f5089d9b700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f508c0ff750 0x7f508c195de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508ad9d700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f508c100150 0x7f508c19ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 -- 172.21.15.78:0/2656189742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f508c100150 msgr2=0x7f508c19ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f508c100150 0x7f508c19ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.183 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 -- 172.21.15.78:0/2656189742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f508c0ff750 msgr2=0x7f508c195de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f508c0ff750 0x7f508c195de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 -- 172.21.15.78:0/2656189742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50800095f0 con 0x7f508c0feda0 2023-07-14T19:04:19.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f508a59c700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 0x7f508c1958a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f507c00d980 tx=0x7f507c00dd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f507b7fe700 1 -- 172.21.15.78:0/2656189742 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c00de60 con 0x7f508c0feda0 2023-07-14T19:04:19.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f5090975700 1 -- 172.21.15.78:0/2656189742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f508c19d9d0 con 0x7f508c0feda0 2023-07-14T19:04:19.184 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f507b7fe700 1 -- 172.21.15.78:0/2656189742 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f507c0044b0 con 0x7f508c0feda0 2023-07-14T19:04:19.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f507b7fe700 1 -- 172.21.15.78:0/2656189742 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c003b40 con 0x7f508c0feda0 2023-07-14T19:04:19.185 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.176+0000 7f5090975700 1 -- 172.21.15.78:0/2656189742 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c19dee0 con 0x7f508c0feda0 2023-07-14T19:04:19.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.179+0000 7f507b7fe700 1 -- 172.21.15.78:0/2656189742 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f507c003ca0 con 0x7f508c0feda0 2023-07-14T19:04:19.192 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.179+0000 7f507b7fe700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50740a1150 0x7f50740a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.180+0000 7f507b7fe700 1 -- 172.21.15.78:0/2656189742 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f507c0cc950 con 0x7f508c0feda0 2023-07-14T19:04:19.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.180+0000 7f5089d9b700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50740a1150 0x7f50740a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.193 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.180+0000 7f5089d9b700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50740a1150 0x7f50740a3600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5080006010 tx=0x7f508000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.180+0000 7f509cb0db80 1 Processor -- start 2023-07-14T19:04:19.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f509cb0db80 1 -- start start 2023-07-14T19:04:19.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dc8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.194 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 0x559d50dc8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dc98d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f509cb0db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50cccc60 con 0x559d50dc9460 2023-07-14T19:04:19.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f509cb0db80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50d1f9f0 con 0x559d50dc7f10 2023-07-14T19:04:19.195 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f509cb0db80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50d18e80 con 0x559d50dc8860 2023-07-14T19:04:19.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dc98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f5089d9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 0x559d50dc8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dc98d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36128/0 (socket says 172.21.15.78:36128) 2023-07-14T19:04:19.196 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 -- 172.21.15.78:0/1099937103 learned_addr learned my addr 172.21.15.78:0/1099937103 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508a59c700 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dc8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 -- 172.21.15.78:0/1099937103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 msgr2=0x559d50dc8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.197 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 0x559d50dc8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 -- 172.21.15.78:0/1099937103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 msgr2=0x559d50dc8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dc8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 -- 172.21.15.78:0/1099937103 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d50b46d90 con 0x559d50dc9460 2023-07-14T19:04:19.198 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508a59c700 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dc8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:19.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f508ad9d700 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dc98d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5084009fd0 tx=0x7f5084009920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d78e244a0bc7fe5 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f5078ff9700 1 -- 172.21.15.78:0/1099937103 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084009e70 con 0x559d50dc9460 2023-07-14T19:04:19.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.181+0000 7f5078ff9700 1 -- 172.21.15.78:0/1099937103 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f508400f050 con 0x559d50dc9460 2023-07-14T19:04:19.199 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f5078ff9700 1 -- 172.21.15.78:0/1099937103 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084014410 con 0x559d50dc9460 2023-07-14T19:04:19.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 -- 172.21.15.78:0/1099937103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 msgr2=0x559d50dc98d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.200 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dc98d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5084009fd0 tx=0x7f5084009920 comp rx=0 tx=0).stop 2023-07-14T19:04:19.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 -- 172.21.15.78:0/1099937103 shutdown_connections 2023-07-14T19:04:19.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dc8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dc98d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.201 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1099937103 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 0x559d50dc8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 -- 172.21.15.78:0/1099937103 >> 172.21.15.78:0/1099937103 conn(0x559d50dbd630 msgr2=0x559d50dc2980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 -- 172.21.15.78:0/1099937103 shutdown_connections 2023-07-14T19:04:19.202 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.182+0000 7f509cb0db80 1 -- 172.21.15.78:0/1099937103 wait complete. 2023-07-14T19:04:19.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 1 -- start start 2023-07-14T19:04:19.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 1 Processor -- start 2023-07-14T19:04:19.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dd18c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.203 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 0x559d50dd1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dd8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50cb8a70 con 0x559d50dc9460 2023-07-14T19:04:19.204 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50cb8770 con 0x559d50dc7f10 2023-07-14T19:04:19.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50b9c8d0 con 0x559d50dc8860 2023-07-14T19:04:19.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f509cb0db80 10 client.0 _ll_register_callbacks cb 0x559d50cb6710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:04:19.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dd8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.205 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508a59c700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dd18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508ad9d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dd8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36134/0 (socket says 172.21.15.78:36134) 2023-07-14T19:04:19.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508ad9d700 1 -- 172.21.15.78:0/1755244041 learned_addr learned my addr 172.21.15.78:0/1755244041 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f5089d9b700 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 0x559d50dd1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.206 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508a59c700 1 -- 172.21.15.78:0/1755244041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 msgr2=0x559d50dd1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508a59c700 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 0x559d50dd1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508a59c700 1 -- 172.21.15.78:0/1755244041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 msgr2=0x559d50dd8e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.207 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508a59c700 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dd8e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508a59c700 1 -- 172.21.15.78:0/1755244041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5084009050 con 0x559d50dc7f10 2023-07-14T19:04:19.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.183+0000 7f508ad9d700 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dd8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:04:19.208 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f508a59c700 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dd18c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f507c009520 tx=0x7f507c009c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a59e2abbcf9f6d0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f5071ffb700 10 client.0 ms_handle_connect on v2:172.21.15.88:3300/0 2023-07-14T19:04:19.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f5071ffb700 1 -- 172.21.15.78:0/1755244041 <== mon.2 v2:172.21.15.88:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c099140 con 0x559d50dc7f10 2023-07-14T19:04:19.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 1 -- 172.21.15.78:0/1755244041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d50c724a0 con 0x559d50dc7f10 2023-07-14T19:04:19.209 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f5071ffb700 1 -- 172.21.15.78:0/1755244041 <== mon.2 v2:172.21.15.88:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f507c0b2210 con 0x559d50dc7f10 2023-07-14T19:04:19.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f5071ffb700 1 -- 172.21.15.78:0/1755244041 <== mon.2 v2:172.21.15.88:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f507c09da80 con 0x559d50dc7f10 2023-07-14T19:04:19.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 10 client.25049 fetch_fsmap learned FSMap version 23 2023-07-14T19:04:19.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 1 -- 172.21.15.78:0/1755244041 --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559d50b2fc10 con 0x559d50dc7f10 2023-07-14T19:04:19.210 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f5071ffb700 1 -- 172.21.15.78:0/1755244041 <== mon.2 v2:172.21.15.88:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x559d50c724a0 con 0x559d50dc7f10 2023-07-14T19:04:19.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f5071ffb700 1 -- 172.21.15.78:0/1755244041 <== mon.2 v2:172.21.15.88:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f507c0989b0 con 0x559d50dc7f10 2023-07-14T19:04:19.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 10 client.25049 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:04:19.211 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 10 client.25049 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:04:19.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 20 client.25049 populate_metadata read hostname 'smithi078' 2023-07-14T19:04:19.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x559d50ddcb00 0x559d50ddbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 4 client.25049 mds_command: new command op to 24485 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:04:19.212 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.184+0000 7f509cb0db80 1 -- 172.21.15.78:0/1755244041 --> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d50d96d50 con 0x559d50ddcb00 2023-07-14T19:04:19.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.185+0000 7f5089d9b700 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x559d50ddcb00 0x559d50ddbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.185+0000 7f5089d9b700 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x559d50ddcb00 0x559d50ddbf40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=168ad9111136cab0 server_cookie=840aac72a156c395 in_seq=0 out_seq=0 2023-07-14T19:04:19.213 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.185+0000 7f5071ffb700 10 client.25049 ms_handle_connect on v2:172.21.15.88:6836/1950772288 2023-07-14T19:04:19.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f5071ffb700 1 -- 172.21.15.78:0/1755244041 <== mds.2 v2:172.21.15.88:6836/1950772288 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x559d50d96d50 con 0x559d50ddcb00 2023-07-14T19:04:19.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f5071ffb700 10 client.25049 handle_command_reply: tid=0 2023-07-14T19:04:19.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 client.25049 shutdown 2023-07-14T19:04:19.214 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 -- 172.21.15.78:0/1755244041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 msgr2=0x559d50dd18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dd18c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f507c009520 tx=0x7f507c009c30 comp rx=0 tx=0).stop 2023-07-14T19:04:19.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 -- 172.21.15.78:0/1755244041 shutdown_connections 2023-07-14T19:04:19.215 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x559d50ddcb00 0x559d50ddbf40 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dd18c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f5071ffb700 0 client.25049 ms_handle_reset on v2:172.21.15.88:6836/1950772288 2023-07-14T19:04:19.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc9460 0x559d50dd8e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.216 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 --2- 172.21.15.78:0/1755244041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc8860 0x559d50dd1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 -- 172.21.15.78:0/1755244041 >> 172.21.15.78:0/1755244041 conn(0x559d50dbd630 msgr2=0x559d50dc1600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 -- 172.21.15.78:0/1755244041 shutdown_connections 2023-07-14T19:04:19.217 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 1 -- 172.21.15.78:0/1755244041 wait complete. 2023-07-14T19:04:19.218 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.186+0000 7f509cb0db80 20 client.25049 trim_cache size 0 max 16384 2023-07-14T19:04:19.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f509cb0db80 1 Processor -- start 2023-07-14T19:04:19.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f509cb0db80 1 -- start start 2023-07-14T19:04:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dcf7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc8860 0x559d50dd8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50ddbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f509cb0db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50ce7790 con 0x559d50dc8860 2023-07-14T19:04:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f509cb0db80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50ba8de0 con 0x559d50dc7f10 2023-07-14T19:04:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f509cb0db80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50cb4d50 con 0x559d50dc9460 2023-07-14T19:04:19.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.203+0000 7f5089d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc8860 0x559d50dd8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f5089d9b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc8860 0x559d50dd8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36140/0 (socket says 172.21.15.78:36140) 2023-07-14T19:04:19.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f5089d9b700 1 -- 172.21.15.78:0/534725495 learned_addr learned my addr 172.21.15.78:0/534725495 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f508a59c700 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dcf7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f508ad9d700 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50ddbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f5089d9b700 1 -- 172.21.15.78:0/534725495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 msgr2=0x559d50ddbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f5089d9b700 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50ddbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f5089d9b700 1 -- 172.21.15.78:0/534725495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 msgr2=0x559d50dcf7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f5089d9b700 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dcf7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f5089d9b700 1 -- 172.21.15.78:0/534725495 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d50b6d200 con 0x559d50dc8860 2023-07-14T19:04:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f508a59c700 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dcf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:19.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f5089d9b700 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc8860 0x559d50dd8e00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5080004360 tx=0x7f508001f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5a1cd8ced2d43e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f50797fa700 1 -- 172.21.15.78:0/534725495 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508002b040 con 0x559d50dc8860 2023-07-14T19:04:19.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f50797fa700 1 -- 172.21.15.78:0/534725495 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f508002c040 con 0x559d50dc8860 2023-07-14T19:04:19.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.204+0000 7f50797fa700 1 -- 172.21.15.78:0/534725495 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5080003b60 con 0x559d50dc8860 2023-07-14T19:04:19.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- 172.21.15.78:0/534725495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc8860 msgr2=0x559d50dd8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc8860 0x559d50dd8e00 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f5080004360 tx=0x7f508001f000 comp rx=0 tx=0).stop 2023-07-14T19:04:19.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- 172.21.15.78:0/534725495 shutdown_connections 2023-07-14T19:04:19.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc7f10 0x559d50dcf7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.230 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50ddbf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 --2- 172.21.15.78:0/534725495 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc8860 0x559d50dd8e00 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- 172.21.15.78:0/534725495 >> 172.21.15.78:0/534725495 conn(0x559d50dbd630 msgr2=0x559d50dd9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- 172.21.15.78:0/534725495 shutdown_connections 2023-07-14T19:04:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- 172.21.15.78:0/534725495 wait complete. 2023-07-14T19:04:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- start start 2023-07-14T19:04:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 Processor -- start 2023-07-14T19:04:19.231 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc7f10 0x559d50dd4070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc8860 0x559d50dd45b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50e30780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50dd4bd0 con 0x559d50dc7f10 2023-07-14T19:04:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50e30cc0 con 0x559d50dc8860 2023-07-14T19:04:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d50e30e00 con 0x559d50dc9460 2023-07-14T19:04:19.232 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f509cb0db80 10 client.0 _ll_register_callbacks cb 0x7f507c0993b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:04:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f508a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc7f10 0x559d50dd4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f508a59c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc7f10 0x559d50dd4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36144/0 (socket says 172.21.15.78:36144) 2023-07-14T19:04:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.205+0000 7f508a59c700 1 -- 172.21.15.78:0/910883086 learned_addr learned my addr 172.21.15.78:0/910883086 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f508ad9d700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50e30780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f508a59c700 1 -- 172.21.15.78:0/910883086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 msgr2=0x559d50e30780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f508a59c700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50e30780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f508a59c700 1 -- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc8860 msgr2=0x559d50dd45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f5089d9b700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc8860 0x559d50dd45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f508a59c700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc8860 0x559d50dd45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f508a59c700 1 -- 172.21.15.78:0/910883086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f508c107870 con 0x559d50dc7f10 2023-07-14T19:04:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f508ad9d700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50e30780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f5089d9b700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc8860 0x559d50dd45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f508a59c700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc7f10 0x559d50dd4070 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f507c0ca190 tx=0x7f507c005c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5890e9f0c19b541c server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f5071ffb700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:04:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f5071ffb700 1 -- 172.21.15.78:0/910883086 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f507c0b2210 con 0x559d50dc7f10 2023-07-14T19:04:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f5071ffb700 1 -- 172.21.15.78:0/910883086 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f507c09da80 con 0x559d50dc7f10 2023-07-14T19:04:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 1 -- 172.21.15.78:0/910883086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f508400f050 con 0x559d50dc7f10 2023-07-14T19:04:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f5071ffb700 1 -- 172.21.15.78:0/910883086 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f507c0a1ea0 con 0x559d50dc7f10 2023-07-14T19:04:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 10 client.15408 fetch_fsmap learned FSMap version 23 2023-07-14T19:04:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 1 -- 172.21.15.78:0/910883086 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x559d50b45260 con 0x559d50dc7f10 2023-07-14T19:04:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f5071ffb700 1 -- 172.21.15.78:0/910883086 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f508400f050 con 0x559d50dc7f10 2023-07-14T19:04:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f5071ffb700 1 -- 172.21.15.78:0/910883086 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f507c084320 con 0x559d50dc7f10 2023-07-14T19:04:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 10 client.15408 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:04:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 10 client.15408 resolve_mds: resolved name 'h' to daemon mds.h 2023-07-14T19:04:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 20 client.15408 populate_metadata read hostname 'smithi078' 2023-07-14T19:04:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x559d50ddcb00 0x559d50e32020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.236 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 4 client.15408 mds_command: new command op to 24485 tid=0[{"prefix": "status"}] 2023-07-14T19:04:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.206+0000 7f509cb0db80 1 -- 172.21.15.78:0/910883086 --> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] -- command(tid 0: {"prefix": "status"}) v1 -- 0x559d50d11e60 con 0x559d50ddcb00 2023-07-14T19:04:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f5089d9b700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x559d50ddcb00 0x559d50e32020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f5071ffb700 10 client.15408 ms_handle_connect on v2:172.21.15.88:6836/1950772288 2023-07-14T19:04:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f5089d9b700 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x559d50ddcb00 0x559d50e32020 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=36bf4a15b63567d1 server_cookie=450af2ed8f7b0e92 in_seq=0 out_seq=0 2023-07-14T19:04:19.237 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f5071ffb700 1 -- 172.21.15.78:0/910883086 <== mds.2 v2:172.21.15.88:6836/1950772288 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x559d50d11e60 con 0x559d50ddcb00 2023-07-14T19:04:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f5071ffb700 10 client.15408 handle_command_reply: tid=0 2023-07-14T19:04:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 client.15408 shutdown 2023-07-14T19:04:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 -- 172.21.15.78:0/910883086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc7f10 msgr2=0x559d50dd4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc7f10 0x559d50dd4070 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f507c0ca190 tx=0x7f507c005c60 comp rx=0 tx=0).stop 2023-07-14T19:04:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 -- 172.21.15.78:0/910883086 shutdown_connections 2023-07-14T19:04:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559d50dc9460 0x559d50e30780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.238 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x559d50dc7f10 0x559d50dd4070 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:6836/1950772288,v1:172.21.15.88:6837/1950772288] conn(0x559d50ddcb00 0x559d50e32020 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 --2- 172.21.15.78:0/910883086 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x559d50dc8860 0x559d50dd45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f5071ffb700 0 client.15408 ms_handle_reset on v2:172.21.15.88:6836/1950772288 2023-07-14T19:04:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 -- 172.21.15.78:0/910883086 >> 172.21.15.78:0/910883086 conn(0x559d50dbd630 msgr2=0x559d50dd9670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 -- 172.21.15.78:0/910883086 shutdown_connections 2023-07-14T19:04:19.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.207+0000 7f509cb0db80 1 -- 172.21.15.78:0/910883086 wait complete. 2023-07-14T19:04:19.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f509cb0db80 20 client.15408 trim_cache size 0 max 16384 2023-07-14T19:04:19.240 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:04:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T19:04:19.240 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 2, 2023-07-14T19:04:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24485, 2023-07-14T19:04:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-07-14T19:04:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-07-14T19:04:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-07-14T19:04:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 89.008536687000003, 2023-07-14T19:04:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2023-07-14T19:04:19.241 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2023-07-14T19:04:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T19:04:19.242 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 123.094776873 2023-07-14T19:04:19.242 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:04:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 -- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50740a1150 msgr2=0x7f50740a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50740a1150 0x7f50740a3600 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f5080006010 tx=0x7f508000b260 comp rx=0 tx=0).stop 2023-07-14T19:04:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 -- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 msgr2=0x7f508c1958a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 0x7f508c1958a0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f507c00d980 tx=0x7f507c00dd40 comp rx=0 tx=0).stop 2023-07-14T19:04:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 -- 172.21.15.78:0/2656189742 shutdown_connections 2023-07-14T19:04:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f50740a1150 0x7f50740a3600 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f508c100150 0x7f508c19ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f508c0feda0 0x7f508c1958a0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 --2- 172.21.15.78:0/2656189742 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f508c0ff750 0x7f508c195de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 -- 172.21.15.78:0/2656189742 >> 172.21.15.78:0/2656189742 conn(0x7f508c0f98e0 msgr2=0x7f508c0fbb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 -- 172.21.15.78:0/2656189742 shutdown_connections 2023-07-14T19:04:19.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.208+0000 7f5090975700 1 -- 172.21.15.78:0/2656189742 wait complete. 2023-07-14T19:04:19.245 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2023-07-14T19:04:19.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.364+0000 7f85e2e67700 1 -- 172.21.15.78:0/1792764249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc1000f0 msgr2=0x7f85dc100500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.364+0000 7f85e2e67700 1 --2- 172.21.15.78:0/1792764249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc1000f0 0x7f85dc100500 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f85d0009990 tx=0x7f85d0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 -- 172.21.15.78:0/1792764249 shutdown_connections 2023-07-14T19:04:19.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 --2- 172.21.15.78:0/1792764249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85dc0ff6f0 0x7f85dc106fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 --2- 172.21.15.78:0/1792764249 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc1000f0 0x7f85dc100500 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 --2- 172.21.15.78:0/1792764249 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0fed40 0x7f85dc0ff1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 -- 172.21.15.78:0/1792764249 >> 172.21.15.78:0/1792764249 conn(0x7f85dc0f98e0 msgr2=0x7f85dc0fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 -- 172.21.15.78:0/1792764249 shutdown_connections 2023-07-14T19:04:19.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 -- 172.21.15.78:0/1792764249 wait complete. 2023-07-14T19:04:19.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 Processor -- start 2023-07-14T19:04:19.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 -- start start 2023-07-14T19:04:19.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0fed40 0x7f85dc195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc0ff6f0 0x7f85dc195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85dc1000f0 0x7f85dc19ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc19d410 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc19d550 con 0x7f85dc0fed40 2023-07-14T19:04:19.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85e2e67700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85dc19d6e0 con 0x7f85dc1000f0 2023-07-14T19:04:19.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85dbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc0ff6f0 0x7f85dc195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85dbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc0ff6f0 0x7f85dc195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36160/0 (socket says 172.21.15.78:36160) 2023-07-14T19:04:19.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.365+0000 7f85dbfff700 1 -- 172.21.15.78:0/3546143919 learned_addr learned my addr 172.21.15.78:0/3546143919 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85e1404700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85dc1000f0 0x7f85dc19ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85e0c03700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0fed40 0x7f85dc195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85dbfff700 1 -- 172.21.15.78:0/3546143919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85dc1000f0 msgr2=0x7f85dc19ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85dbfff700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85dc1000f0 0x7f85dc19ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85dbfff700 1 -- 172.21.15.78:0/3546143919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0fed40 msgr2=0x7f85dc195890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85dbfff700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0fed40 0x7f85dc195890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85dbfff700 1 -- 172.21.15.78:0/3546143919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d00095f0 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85e0c03700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0fed40 0x7f85dc195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:19.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85dbfff700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc0ff6f0 0x7f85dc195dd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f85c800e9a0 tx=0x7f85c800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85d9ffb700 1 -- 172.21.15.78:0/3546143919 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8009790 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85d9ffb700 1 -- 172.21.15.78:0/3546143919 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f85c800cbd0 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85e2e67700 1 -- 172.21.15.78:0/3546143919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85dc19d9c0 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85d9ffb700 1 -- 172.21.15.78:0/3546143919 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8018680 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.366+0000 7f85e2e67700 1 -- 172.21.15.78:0/3546143919 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc19df20 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.368+0000 7f85d9ffb700 1 -- 172.21.15.78:0/3546143919 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f85c800cd40 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.369+0000 7f85d9ffb700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85cc0a11d0 0x7f85cc0a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.369+0000 7f85d9ffb700 1 -- 172.21.15.78:0/3546143919 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f85c8014070 con 0x7f85dc0ff6f0 2023-07-14T19:04:19.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.369+0000 7f85e0c03700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85cc0a11d0 0x7f85cc0a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.369+0000 7f85e0c03700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85cc0a11d0 0x7f85cc0a3680 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f85d00059d0 tx=0x7f85d0005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85eefffb80 1 Processor -- start 2023-07-14T19:04:19.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85eefffb80 1 -- start start 2023-07-14T19:04:19.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 0x55e61720b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e61720be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85eefffb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61710fc60 con 0x55e61720b860 2023-07-14T19:04:19.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85eefffb80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6171629f0 con 0x55e61720c460 2023-07-14T19:04:19.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85eefffb80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61715be80 con 0x55e61720af10 2023-07-14T19:04:19.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85dbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e61720be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85dbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e61720be80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36168/0 (socket says 172.21.15.78:36168) 2023-07-14T19:04:19.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85dbfff700 1 -- 172.21.15.78:0/368974962 learned_addr learned my addr 172.21.15.78:0/368974962 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85e0c03700 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 0x55e61720b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85e1404700 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85dbfff700 1 -- 172.21.15.78:0/368974962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 msgr2=0x55e61720b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85dbfff700 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 0x55e61720b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.370+0000 7f85dbfff700 1 -- 172.21.15.78:0/368974962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 msgr2=0x55e61720c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85dbfff700 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85dbfff700 1 -- 172.21.15.78:0/368974962 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e616f89d90 con 0x55e61720b860 2023-07-14T19:04:19.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85e1404700 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:19.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85dbfff700 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e61720be80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f85c8096440 tx=0x7f85c8099b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cb51589965f4dff server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85c6ffd700 1 -- 172.21.15.78:0/368974962 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8099f40 con 0x55e61720b860 2023-07-14T19:04:19.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85c6ffd700 1 -- 172.21.15.78:0/368974962 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f85c809df70 con 0x55e61720b860 2023-07-14T19:04:19.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85c6ffd700 1 -- 172.21.15.78:0/368974962 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c80a2140 con 0x55e61720b860 2023-07-14T19:04:19.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85eefffb80 1 -- 172.21.15.78:0/368974962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 msgr2=0x55e61720be80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85eefffb80 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e61720be80 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f85c8096440 tx=0x7f85c8099b50 comp rx=0 tx=0).stop 2023-07-14T19:04:19.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85eefffb80 1 -- 172.21.15.78:0/368974962 shutdown_connections 2023-07-14T19:04:19.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85eefffb80 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85eefffb80 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 0x55e61720b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85eefffb80 1 --2- 172.21.15.78:0/368974962 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e61720be80 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.371+0000 7f85eefffb80 1 -- 172.21.15.78:0/368974962 >> 172.21.15.78:0/368974962 conn(0x55e617200630 msgr2=0x55e617205980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 -- 172.21.15.78:0/368974962 shutdown_connections 2023-07-14T19:04:19.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 -- 172.21.15.78:0/368974962 wait complete. 2023-07-14T19:04:19.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 -- start start 2023-07-14T19:04:19.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 Processor -- start 2023-07-14T19:04:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720af10 0x55e617214940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720b860 0x55e617214e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720c460 0x55e61721bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6170fba70 con 0x55e61720c460 2023-07-14T19:04:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6170fb770 con 0x55e61720af10 2023-07-14T19:04:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e616fdf8d0 con 0x55e61720b860 2023-07-14T19:04:19.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85eefffb80 10 client.0 _ll_register_callbacks cb 0x55e6170f9710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:04:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85e1404700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720c460 0x55e61721bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85e1404700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720c460 0x55e61721bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36176/0 (socket says 172.21.15.78:36176) 2023-07-14T19:04:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.372+0000 7f85e1404700 1 -- 172.21.15.78:0/3195148411 learned_addr learned my addr 172.21.15.78:0/3195148411 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85dbfff700 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720b860 0x55e617214e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85e0c03700 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720af10 0x55e617214940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85e1404700 1 -- 172.21.15.78:0/3195148411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720b860 msgr2=0x55e617214e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85e1404700 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720b860 0x55e617214e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85e1404700 1 -- 172.21.15.78:0/3195148411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720af10 msgr2=0x55e617214940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85e1404700 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720af10 0x55e617214940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85e1404700 1 -- 172.21.15.78:0/3195148411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85dc19df20 con 0x55e61720c460 2023-07-14T19:04:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85e1404700 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720c460 0x55e61721bec0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f85d4007c60 tx=0x7f85d4007c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4fa32c6f875c3e server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85bbfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:04:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85bbfff700 1 -- 172.21.15.78:0/3195148411 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d400e040 con 0x55e61720c460 2023-07-14T19:04:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85bbfff700 1 -- 172.21.15.78:0/3195148411 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f85d4013b30 con 0x55e61720c460 2023-07-14T19:04:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85eefffb80 1 -- 172.21.15.78:0/3195148411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55e6170b54a0 con 0x55e61720c460 2023-07-14T19:04:19.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85bbfff700 1 -- 172.21.15.78:0/3195148411 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85d4013d30 con 0x55e61720c460 2023-07-14T19:04:19.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85eefffb80 10 client.15420 fetch_fsmap learned FSMap version 23 2023-07-14T19:04:19.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85eefffb80 1 -- 172.21.15.78:0/3195148411 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e616f72c10 con 0x55e61720c460 2023-07-14T19:04:19.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85bbfff700 1 -- 172.21.15.78:0/3195148411 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x55e6170b54a0 con 0x55e61720c460 2023-07-14T19:04:19.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.373+0000 7f85bbfff700 1 -- 172.21.15.78:0/3195148411 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f85d401e3b0 con 0x55e61720c460 2023-07-14T19:04:19.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85eefffb80 10 client.15420 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:04:19.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85eefffb80 10 client.15420 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:04:19.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85eefffb80 20 client.15420 populate_metadata read hostname 'smithi078' 2023-07-14T19:04:19.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85eefffb80 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x55e61721fc80 0x55e61721efd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85eefffb80 4 client.15420 mds_command: new command op to 24487 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:04:19.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85eefffb80 1 -- 172.21.15.78:0/3195148411 --> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6171d9d50 con 0x55e61721fc80 2023-07-14T19:04:19.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85e0c03700 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x55e61721fc80 0x55e61721efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85bbfff700 10 client.15420 ms_handle_connect on v2:172.21.15.102:6838/984471622 2023-07-14T19:04:19.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.374+0000 7f85e0c03700 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x55e61721fc80 0x55e61721efd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=237b475ad726975a server_cookie=b017a6940116a499 in_seq=0 out_seq=0 2023-07-14T19:04:19.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85bbfff700 1 -- 172.21.15.78:0/3195148411 <== mds.1 v2:172.21.15.102:6838/984471622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x55e6171d9d50 con 0x55e61721fc80 2023-07-14T19:04:19.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85bbfff700 10 client.15420 handle_command_reply: tid=0 2023-07-14T19:04:19.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 client.15420 shutdown 2023-07-14T19:04:19.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 -- 172.21.15.78:0/3195148411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720c460 msgr2=0x55e61721bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720c460 0x55e61721bec0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f85d4007c60 tx=0x7f85d4007c90 comp rx=0 tx=0).stop 2023-07-14T19:04:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 -- 172.21.15.78:0/3195148411 shutdown_connections 2023-07-14T19:04:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x55e61721fc80 0x55e61721efd0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720af10 0x55e617214940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720c460 0x55e61721bec0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85bbfff700 0 client.15420 ms_handle_reset on v2:172.21.15.102:6838/984471622 2023-07-14T19:04:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 --2- 172.21.15.78:0/3195148411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720b860 0x55e617214e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 -- 172.21.15.78:0/3195148411 >> 172.21.15.78:0/3195148411 conn(0x55e617200630 msgr2=0x55e617204600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 -- 172.21.15.78:0/3195148411 shutdown_connections 2023-07-14T19:04:19.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 1 -- 172.21.15.78:0/3195148411 wait complete. 2023-07-14T19:04:19.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.375+0000 7f85eefffb80 20 client.15420 trim_cache size 0 max 16384 2023-07-14T19:04:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85eefffb80 1 Processor -- start 2023-07-14T19:04:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85eefffb80 1 -- start start 2023-07-14T19:04:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 0x55e6172050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e6172127e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85eefffb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6170b54d0 con 0x55e61720b860 2023-07-14T19:04:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85eefffb80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61712a7c0 con 0x55e61720c460 2023-07-14T19:04:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85eefffb80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e61711a480 con 0x55e61720af10 2023-07-14T19:04:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e6172127e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e6172127e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36182/0 (socket says 172.21.15.78:36182) 2023-07-14T19:04:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 -- 172.21.15.78:0/569869565 learned_addr learned my addr 172.21.15.78:0/569869565 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85e1404700 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 -- 172.21.15.78:0/569869565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 msgr2=0x55e6172050a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 0x55e6172050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 -- 172.21.15.78:0/569869565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 msgr2=0x55e61720f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 -- 172.21.15.78:0/569869565 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e617205f00 con 0x55e61720b860 2023-07-14T19:04:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85e1404700 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.393+0000 7f85dbfff700 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e6172127e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f85c80ca9a0 tx=0x7f85c801f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78158a053bdc7983 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85c77fe700 1 -- 172.21.15.78:0/569869565 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c8099f40 con 0x55e61720b860 2023-07-14T19:04:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85c77fe700 1 -- 172.21.15.78:0/569869565 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f85c80a2390 con 0x55e61720b860 2023-07-14T19:04:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85c77fe700 1 -- 172.21.15.78:0/569869565 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c80b8330 con 0x55e61720b860 2023-07-14T19:04:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 -- 172.21.15.78:0/569869565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 msgr2=0x55e6172127e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e6172127e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f85c80ca9a0 tx=0x7f85c801f000 comp rx=0 tx=0).stop 2023-07-14T19:04:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 -- 172.21.15.78:0/569869565 shutdown_connections 2023-07-14T19:04:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720c460 0x55e61720f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720af10 0x55e6172050a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 --2- 172.21.15.78:0/569869565 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720b860 0x55e6172127e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 -- 172.21.15.78:0/569869565 >> 172.21.15.78:0/569869565 conn(0x55e617200630 msgr2=0x55e617213620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 -- 172.21.15.78:0/569869565 shutdown_connections 2023-07-14T19:04:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 -- 172.21.15.78:0/569869565 wait complete. 2023-07-14T19:04:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 -- start start 2023-07-14T19:04:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.394+0000 7f85eefffb80 1 Processor -- start 2023-07-14T19:04:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720af10 0x55e6172738c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720b860 0x55e617275e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85eefffb80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720c460 0x55e617278360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85eefffb80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e617278a20 con 0x55e61720af10 2023-07-14T19:04:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85eefffb80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e617278b90 con 0x55e61720b860 2023-07-14T19:04:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85eefffb80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e617278d50 con 0x55e61720c460 2023-07-14T19:04:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85eefffb80 10 client.0 _ll_register_callbacks cb 0x7f85d40075e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:04:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720af10 0x55e6172738c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720af10 0x55e6172738c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36186/0 (socket says 172.21.15.78:36186) 2023-07-14T19:04:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 -- 172.21.15.78:0/1583418072 learned_addr learned my addr 172.21.15.78:0/1583418072 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e1404700 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720c460 0x55e617278360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85dbfff700 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720b860 0x55e617275e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 -- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720c460 msgr2=0x55e617278360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720c460 0x55e617278360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 -- 172.21.15.78:0/1583418072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720b860 msgr2=0x55e617275e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720b860 0x55e617275e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 -- 172.21.15.78:0/1583418072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e616f89d90 con 0x55e61720af10 2023-07-14T19:04:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e1404700 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720c460 0x55e617278360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85e0c03700 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720af10 0x55e6172738c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f85d0006010 tx=0x7f85d0009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3c2d588299916e4 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.395+0000 7f85bbfff700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:04:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85bbfff700 1 -- 172.21.15.78:0/1583418072 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85d002b040 con 0x55e61720af10 2023-07-14T19:04:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 1 -- 172.21.15.78:0/1583418072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f85c809df70 con 0x55e61720af10 2023-07-14T19:04:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85bbfff700 1 -- 172.21.15.78:0/1583418072 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f85d0034110 con 0x55e61720af10 2023-07-14T19:04:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85bbfff700 1 -- 172.21.15.78:0/1583418072 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f85d00342d0 con 0x55e61720af10 2023-07-14T19:04:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 10 client.15426 fetch_fsmap learned FSMap version 23 2023-07-14T19:04:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 1 -- 172.21.15.78:0/1583418072 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x55e61721c0b0 con 0x55e61720af10 2023-07-14T19:04:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85bbfff700 1 -- 172.21.15.78:0/1583418072 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f85c809df70 con 0x55e61720af10 2023-07-14T19:04:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85bbfff700 1 -- 172.21.15.78:0/1583418072 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7f85d0005300 con 0x55e61720af10 2023-07-14T19:04:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 10 client.15426 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:04:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 10 client.15426 resolve_mds: resolved name 'i' to daemon mds.i 2023-07-14T19:04:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 20 client.15426 populate_metadata read hostname 'smithi078' 2023-07-14T19:04:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x55e61727daf0 0x55e61727a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 4 client.15426 mds_command: new command op to 24487 tid=0[{"prefix": "status"}] 2023-07-14T19:04:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85eefffb80 1 -- 172.21.15.78:0/1583418072 --> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e617154e60 con 0x55e61727daf0 2023-07-14T19:04:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.396+0000 7f85dbfff700 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x55e61727daf0 0x55e61727a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85bbfff700 10 client.15426 ms_handle_connect on v2:172.21.15.102:6838/984471622 2023-07-14T19:04:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85dbfff700 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x55e61727daf0 0x55e61727a4d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8973d93e03a9de75 server_cookie=3502373274e385c5 in_seq=0 out_seq=0 2023-07-14T19:04:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85bbfff700 1 -- 172.21.15.78:0/1583418072 <== mds.1 v2:172.21.15.102:6838/984471622 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x55e617154e60 con 0x55e61727daf0 2023-07-14T19:04:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85bbfff700 10 client.15426 handle_command_reply: tid=0 2023-07-14T19:04:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 client.15426 shutdown 2023-07-14T19:04:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 -- 172.21.15.78:0/1583418072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720af10 msgr2=0x55e6172738c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720af10 0x55e6172738c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f85d0006010 tx=0x7f85d0009310 comp rx=0 tx=0).stop 2023-07-14T19:04:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 -- 172.21.15.78:0/1583418072 shutdown_connections 2023-07-14T19:04:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e61720c460 0x55e617278360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x55e61720af10 0x55e6172738c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.102:6838/984471622,v1:172.21.15.102:6839/984471622] conn(0x55e61727daf0 0x55e61727a4d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 --2- 172.21.15.78:0/1583418072 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x55e61720b860 0x55e617275e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85bbfff700 0 client.15426 ms_handle_reset on v2:172.21.15.102:6838/984471622 2023-07-14T19:04:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 -- 172.21.15.78:0/1583418072 >> 172.21.15.78:0/1583418072 conn(0x55e617200630 msgr2=0x55e6172050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 -- 172.21.15.78:0/1583418072 shutdown_connections 2023-07-14T19:04:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 1 -- 172.21.15.78:0/1583418072 wait complete. 2023-07-14T19:04:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.397+0000 7f85eefffb80 20 client.15426 trim_cache size 0 max 16384 2023-07-14T19:04:19.417 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:04:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T19:04:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 1, 2023-07-14T19:04:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24487, 2023-07-14T19:04:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-07-14T19:04:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-07-14T19:04:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-07-14T19:04:19.418 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 90.211098774000007, 2023-07-14T19:04:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2023-07-14T19:04:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2023-07-14T19:04:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T19:04:19.419 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 121.14077450400001 2023-07-14T19:04:19.420 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:04:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 -- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85cc0a11d0 msgr2=0x7f85cc0a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85cc0a11d0 0x7f85cc0a3680 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f85d00059d0 tx=0x7f85d0005940 comp rx=0 tx=0).stop 2023-07-14T19:04:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 -- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc0ff6f0 msgr2=0x7f85dc195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc0ff6f0 0x7f85dc195dd0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f85c800e9a0 tx=0x7f85c800ecb0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 -- 172.21.15.78:0/3546143919 shutdown_connections 2023-07-14T19:04:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f85cc0a11d0 0x7f85cc0a3680 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f85dc0fed40 0x7f85dc195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f85dc1000f0 0x7f85dc19ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 --2- 172.21.15.78:0/3546143919 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f85dc0ff6f0 0x7f85dc195dd0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 -- 172.21.15.78:0/3546143919 >> 172.21.15.78:0/3546143919 conn(0x7f85dc0f98e0 msgr2=0x7f85dc0fdc30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 -- 172.21.15.78:0/3546143919 shutdown_connections 2023-07-14T19:04:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.398+0000 7f85e2e67700 1 -- 172.21.15.78:0/3546143919 wait complete. 2023-07-14T19:04:19.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:19 smithi078 ceph-mon[127425]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-14T19:04:19.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:19 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1874159743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:19.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:19 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1566172531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:19.423 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:19 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/4173388079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:19.424 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-07-14T19:04:19.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 -- 172.21.15.78:0/3360021377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 msgr2=0x7fc3bc101280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 --2- 172.21.15.78:0/3360021377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 0x7fc3bc101280 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac009990 tx=0x7fc3ac009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 -- 172.21.15.78:0/3360021377 shutdown_connections 2023-07-14T19:04:19.547 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 --2- 172.21.15.78:0/3360021377 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3bc0ffc40 0x7fc3bc100050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 --2- 172.21.15.78:0/3360021377 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3bc1017c0 0x7fc3bc108080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 --2- 172.21.15.78:0/3360021377 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 0x7fc3bc101280 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 -- 172.21.15.78:0/3360021377 >> 172.21.15.78:0/3360021377 conn(0x7fc3bc0fb240 msgr2=0x7fc3bc0fd690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 -- 172.21.15.78:0/3360021377 shutdown_connections 2023-07-14T19:04:19.548 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.543+0000 7fc3c245f700 1 -- 172.21.15.78:0/3360021377 wait complete. 2023-07-14T19:04:19.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c245f700 1 Processor -- start 2023-07-14T19:04:19.549 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c245f700 1 -- start start 2023-07-14T19:04:19.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c245f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3bc0ffc40 0x7fc3bc0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.551 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c245f700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 0x7fc3bc072a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c245f700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3bc1017c0 0x7fc3bc072f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c245f700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc073710 con 0x7fc3bc100e30 2023-07-14T19:04:19.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c245f700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc073880 con 0x7fc3bc1017c0 2023-07-14T19:04:19.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c245f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc19f860 con 0x7fc3bc0ffc40 2023-07-14T19:04:19.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 0x7fc3bc072a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 0x7fc3bc072a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36202/0 (socket says 172.21.15.78:36202) 2023-07-14T19:04:19.552 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3bb7fe700 1 -- 172.21.15.78:0/2460687582 learned_addr learned my addr 172.21.15.78:0/2460687582 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3c09fc700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3bc1017c0 0x7fc3bc072f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.544+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3bc0ffc40 0x7fc3bc0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3bb7fe700 1 -- 172.21.15.78:0/2460687582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3bc0ffc40 msgr2=0x7fc3bc0743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3bc0ffc40 0x7fc3bc0743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3bb7fe700 1 -- 172.21.15.78:0/2460687582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3bc1017c0 msgr2=0x7fc3bc072f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3bc1017c0 0x7fc3bc072f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3bb7fe700 1 -- 172.21.15.78:0/2460687582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3ac0095f0 con 0x7fc3bc100e30 2023-07-14T19:04:19.553 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3bc0ffc40 0x7fc3bc0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:19.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 0x7fc3bc072a30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac005f50 tx=0x7fc3ac0047a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3b97fa700 1 -- 172.21.15.78:0/2460687582 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac01d070 con 0x7fc3bc100e30 2023-07-14T19:04:19.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3c245f700 1 -- 172.21.15.78:0/2460687582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3bc19fae0 con 0x7fc3bc100e30 2023-07-14T19:04:19.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3b97fa700 1 -- 172.21.15.78:0/2460687582 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc3ac004810 con 0x7fc3bc100e30 2023-07-14T19:04:19.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3b97fa700 1 -- 172.21.15.78:0/2460687582 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac005340 con 0x7fc3bc100e30 2023-07-14T19:04:19.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.545+0000 7fc3c245f700 1 -- 172.21.15.78:0/2460687582 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3bc19ff00 con 0x7fc3bc100e30 2023-07-14T19:04:19.554 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.547+0000 7fc3b97fa700 1 -- 172.21.15.78:0/2460687582 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc3ac004980 con 0x7fc3bc100e30 2023-07-14T19:04:19.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.548+0000 7fc3b97fa700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a1150 0x7fc3a40a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.548+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a1150 0x7fc3a40a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.548+0000 7fc3b97fa700 1 -- 172.21.15.78:0/2460687582 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc3ac0cde70 con 0x7fc3bc100e30 2023-07-14T19:04:19.555 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.548+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a1150 0x7fc3a40a3600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0005fd0 tx=0x7fc3b0005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3ce5f7b80 1 Processor -- start 2023-07-14T19:04:19.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3ce5f7b80 1 -- start start 2023-07-14T19:04:19.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d4f10 0x556cbc9d5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.560 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d5860 0x556cbc9d5e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9d68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc8d9c60 con 0x556cbc9d5860 2023-07-14T19:04:19.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc92c9f0 con 0x556cbc9d6460 2023-07-14T19:04:19.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc925e80 con 0x556cbc9d4f10 2023-07-14T19:04:19.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d5860 0x556cbc9d5e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bb7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d5860 0x556cbc9d5e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36210/0 (socket says 172.21.15.78:36210) 2023-07-14T19:04:19.561 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bb7fe700 1 -- 172.21.15.78:0/3217439228 learned_addr learned my addr 172.21.15.78:0/3217439228 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3c09fc700 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9d68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d4f10 0x556cbc9d5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bb7fe700 1 -- 172.21.15.78:0/3217439228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d4f10 msgr2=0x556cbc9d5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d4f10 0x556cbc9d5320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bb7fe700 1 -- 172.21.15.78:0/3217439228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 msgr2=0x556cbc9d68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9d68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.562 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3bb7fe700 1 -- 172.21.15.78:0/3217439228 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cbc753d90 con 0x556cbc9d5860 2023-07-14T19:04:19.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.549+0000 7fc3c09fc700 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9d68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:19.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.550+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d5860 0x556cbc9d5e80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac09af90 tx=0x7fc3ac09b0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2e08989fb4c1427 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.550+0000 7fc3a27fc700 1 -- 172.21.15.78:0/3217439228 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0993a0 con 0x556cbc9d5860 2023-07-14T19:04:19.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.550+0000 7fc3a27fc700 1 -- 172.21.15.78:0/3217439228 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc3ac09b260 con 0x556cbc9d5860 2023-07-14T19:04:19.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.550+0000 7fc3a27fc700 1 -- 172.21.15.78:0/3217439228 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0a3150 con 0x556cbc9d5860 2023-07-14T19:04:19.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.550+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/3217439228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d5860 msgr2=0x556cbc9d5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.563 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.550+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d5860 0x556cbc9d5e80 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac09af90 tx=0x7fc3ac09b0d0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/3217439228 shutdown_connections 2023-07-14T19:04:19.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9d68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d4f10 0x556cbc9d5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/3217439228 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d5860 0x556cbc9d5e80 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/3217439228 >> 172.21.15.78:0/3217439228 conn(0x556cbc9ca630 msgr2=0x556cbc9cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.564 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/3217439228 shutdown_connections 2023-07-14T19:04:19.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/3217439228 wait complete. 2023-07-14T19:04:19.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 -- start start 2023-07-14T19:04:19.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 Processor -- start 2023-07-14T19:04:19.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9de5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9deae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 0x556cbc9e5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc8c5a70 con 0x556cbc9d4f10 2023-07-14T19:04:19.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc8c5770 con 0x556cbc9d5860 2023-07-14T19:04:19.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.551+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc7a98d0 con 0x556cbc9d6460 2023-07-14T19:04:19.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3ce5f7b80 10 client.0 _ll_register_callbacks cb 0x556cbc8c3710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:04:19.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9de5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9de5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36214/0 (socket says 172.21.15.78:36214) 2023-07-14T19:04:19.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2157508814 learned_addr learned my addr 172.21.15.78:0/2157508814 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9deae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3c09fc700 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 0x556cbc9e5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2157508814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 msgr2=0x556cbc9e5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 0x556cbc9e5b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 msgr2=0x556cbc9deae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9deae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2157508814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3bc19ff00 con 0x556cbc9d4f10 2023-07-14T19:04:19.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9deae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:04:19.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9de5a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0011720 tx=0x7fc3b0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cc52f637841657d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.567 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc39b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:04:19.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2157508814 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b001e820 con 0x556cbc9d4f10 2023-07-14T19:04:19.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2157508814 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc3b001e980 con 0x556cbc9d4f10 2023-07-14T19:04:19.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2157508814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556cbc87f4a0 con 0x556cbc9d4f10 2023-07-14T19:04:19.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2157508814 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3b001eb40 con 0x556cbc9d4f10 2023-07-14T19:04:19.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3ce5f7b80 10 client.15438 fetch_fsmap learned FSMap version 23 2023-07-14T19:04:19.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.552+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2157508814 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556cbc73cc10 con 0x556cbc9d4f10 2023-07-14T19:04:19.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2157508814 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x556cbc87f4a0 con 0x556cbc9d4f10 2023-07-14T19:04:19.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2157508814 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc3b0010ba0 con 0x556cbc9d4f10 2023-07-14T19:04:19.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc3ce5f7b80 10 client.15438 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:04:19.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc3ce5f7b80 10 client.15438 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:04:19.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc3ce5f7b80 20 client.15438 populate_metadata read hostname 'smithi078' 2023-07-14T19:04:19.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x556cbc9e98d0 0x556cbc9e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc3ce5f7b80 4 client.15438 mds_command: new command op to 24503 tid=0[{"prefix": "get_command_descriptions"}] 2023-07-14T19:04:19.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2157508814 --> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556cbc9a3d50 con 0x556cbc9e98d0 2023-07-14T19:04:19.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x556cbc9e98d0 0x556cbc9e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc39b7fe700 10 client.15438 ms_handle_connect on v2:172.21.15.88:6838/1379790346 2023-07-14T19:04:19.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.553+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x556cbc9e98d0 0x556cbc9e8c80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=998c39eda42ff1d8 server_cookie=d32049568cdfdf8e in_seq=0 out_seq=0 2023-07-14T19:04:19.571 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2157508814 <== mds.0 v2:172.21.15.88:6838/1379790346 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18017 (crc 0 0 0) 0x556cbc9a3d50 con 0x556cbc9e98d0 2023-07-14T19:04:19.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc39b7fe700 10 client.15438 handle_command_reply: tid=0 2023-07-14T19:04:19.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 client.15438 shutdown 2023-07-14T19:04:19.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2157508814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 msgr2=0x556cbc9de5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9de5a0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0011720 tx=0x7fc3b0004750 comp rx=0 tx=0).stop 2023-07-14T19:04:19.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2157508814 shutdown_connections 2023-07-14T19:04:19.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x556cbc9e98d0 0x556cbc9e8c80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.572 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 0x556cbc9e5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc39b7fe700 0 client.15438 ms_handle_reset on v2:172.21.15.88:6838/1379790346 2023-07-14T19:04:19.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9de5a0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2157508814 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9deae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2157508814 >> 172.21.15.78:0/2157508814 conn(0x556cbc9ca630 msgr2=0x556cbc9ce600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2157508814 shutdown_connections 2023-07-14T19:04:19.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2157508814 wait complete. 2023-07-14T19:04:19.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.554+0000 7fc3ce5f7b80 20 client.15438 trim_cache size 0 max 16384 2023-07-14T19:04:19.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3ce5f7b80 1 Processor -- start 2023-07-14T19:04:19.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3ce5f7b80 1 -- start start 2023-07-14T19:04:19.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e8c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9e91c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 0x556cbc9e5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc92c9f0 con 0x556cbc9d4f10 2023-07-14T19:04:19.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc9e5ff0 con 0x556cbc9d5860 2023-07-14T19:04:19.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc9a3d50 con 0x556cbc9d6460 2023-07-14T19:04:19.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36222/0 (socket says 172.21.15.78:36222) 2023-07-14T19:04:19.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2115358996 learned_addr learned my addr 172.21.15.78:0/2115358996 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2115358996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 msgr2=0x556cbc9e5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 0x556cbc9e5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2115358996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 msgr2=0x556cbc9e91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9e91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9e91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2115358996 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556cbc77a200 con 0x556cbc9d4f10 2023-07-14T19:04:19.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9e91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:19.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e8c80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000efb0 tx=0x7fc3b00263b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8040c8f039c3c67 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3a2ffd700 1 -- 172.21.15.78:0/2115358996 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0026500 con 0x556cbc9d4f10 2023-07-14T19:04:19.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3a2ffd700 1 -- 172.21.15.78:0/2115358996 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc3b0025360 con 0x556cbc9d4f10 2023-07-14T19:04:19.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.572+0000 7fc3a2ffd700 1 -- 172.21.15.78:0/2115358996 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0025970 con 0x556cbc9d4f10 2023-07-14T19:04:19.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2115358996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 msgr2=0x556cbc9e8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e8c80 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000efb0 tx=0x7fc3b00263b0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2115358996 shutdown_connections 2023-07-14T19:04:19.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d6460 0x556cbc9e5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e8c80 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2115358996 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d5860 0x556cbc9e91c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2115358996 >> 172.21.15.78:0/2115358996 conn(0x556cbc9ca630 msgr2=0x556cbc9dc770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.581 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2115358996 shutdown_connections 2023-07-14T19:04:19.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2115358996 wait complete. 2023-07-14T19:04:19.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.573+0000 7fc3ce5f7b80 1 -- start start 2023-07-14T19:04:19.582 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3ce5f7b80 1 Processor -- start 2023-07-14T19:04:19.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d5860 0x556cbc9e11f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3ce5f7b80 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9e4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc9e45f0 con 0x556cbc9d4f10 2023-07-14T19:04:19.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc9e4760 con 0x556cbc9d6460 2023-07-14T19:04:19.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3ce5f7b80 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556cbc9e4920 con 0x556cbc9d5860 2023-07-14T19:04:19.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3ce5f7b80 10 client.0 _ll_register_callbacks cb 0x7fc3b000edf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-07-14T19:04:19.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36228/0 (socket says 172.21.15.78:36228) 2023-07-14T19:04:19.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2618923658 learned_addr learned my addr 172.21.15.78:0/2618923658 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:19.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3c09fc700 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9e4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d5860 0x556cbc9e11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2618923658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d5860 msgr2=0x556cbc9e11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d5860 0x556cbc9e11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2618923658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 msgr2=0x556cbc9e4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-07-14T19:04:19.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9e4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 -- 172.21.15.78:0/2618923658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556cbc73cc10 con 0x556cbc9d4f10 2023-07-14T19:04:19.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc3bbfff700 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e0cb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0026ed0 tx=0x7fc3b0011f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a44fc884534e3d server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:19.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.574+0000 7fc39b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.78:3300/0 2023-07-14T19:04:19.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2618923658 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0026b50 con 0x556cbc9d4f10 2023-07-14T19:04:19.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2618923658 <== mon.0 v2:172.21.15.78:3300/0 2 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc3b001e620 con 0x556cbc9d4f10 2023-07-14T19:04:19.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2618923658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc3ac09b260 con 0x556cbc9d4f10 2023-07-14T19:04:19.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2618923658 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc3b001e820 con 0x556cbc9d4f10 2023-07-14T19:04:19.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 10 client.15444 fetch_fsmap learned FSMap version 23 2023-07-14T19:04:19.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2618923658 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) v3 -- 0x556cbc752260 con 0x556cbc9d4f10 2023-07-14T19:04:19.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2618923658 <== mon.0 v2:172.21.15.78:3300/0 4 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc3ac09b260 con 0x556cbc9d4f10 2023-07-14T19:04:19.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2618923658 <== mon.0 v2:172.21.15.78:3300/0 5 ==== fsmap(e 23) v1 ==== 7470+0+0 (secure 0 0 0) 0x7fc3b0010a60 con 0x556cbc9d4f10 2023-07-14T19:04:19.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 10 client.15444 fetch_fsmap finished waiting for FSMap version 23 2023-07-14T19:04:19.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 10 client.15444 resolve_mds: resolved name 'b' to daemon mds.b 2023-07-14T19:04:19.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 20 client.15444 populate_metadata read hostname 'smithi078' 2023-07-14T19:04:19.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x556cbca425d0 0x556cbca3f520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:19.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 4 client.15444 mds_command: new command op to 24503 tid=0[{"prefix": "status"}] 2023-07-14T19:04:19.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2618923658 --> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556cbc91ee60 con 0x556cbca425d0 2023-07-14T19:04:19.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.575+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x556cbca425d0 0x556cbca3f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:19.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3bb7fe700 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x556cbca425d0 0x556cbca3f520 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce7360f8b7fc9596 server_cookie=93d450f660cfedef in_seq=0 out_seq=0 2023-07-14T19:04:19.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc39b7fe700 10 client.15444 ms_handle_connect on v2:172.21.15.88:6838/1379790346 2023-07-14T19:04:19.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc39b7fe700 1 -- 172.21.15.78:0/2618923658 <== mds.0 v2:172.21.15.88:6838/1379790346 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+326 (crc 0 0 0) 0x556cbc91ee60 con 0x556cbca425d0 2023-07-14T19:04:19.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc39b7fe700 10 client.15444 handle_command_reply: tid=0 2023-07-14T19:04:19.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 client.15444 shutdown 2023-07-14T19:04:19.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2618923658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 msgr2=0x556cbc9e0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e0cb0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0026ed0 tx=0x7fc3b0011f40 comp rx=0 tx=0).stop 2023-07-14T19:04:19.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2618923658 shutdown_connections 2023-07-14T19:04:19.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.88:6838/1379790346,v1:172.21.15.88:6839/1379790346] conn(0x556cbca425d0 0x556cbca3f520 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x556cbc9d6460 0x556cbc9e4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x556cbc9d4f10 0x556cbc9e0cb0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc39b7fe700 0 client.15444 ms_handle_reset on v2:172.21.15.88:6838/1379790346 2023-07-14T19:04:19.591 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 --2- 172.21.15.78:0/2618923658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556cbc9d5860 0x556cbc9e11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2618923658 >> 172.21.15.78:0/2618923658 conn(0x556cbc9ca630 msgr2=0x556cbc9e8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2618923658 shutdown_connections 2023-07-14T19:04:19.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 1 -- 172.21.15.78:0/2618923658 wait complete. 2023-07-14T19:04:19.592 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.576+0000 7fc3ce5f7b80 20 client.15444 trim_cache size 0 max 16384 2023-07-14T19:04:19.593 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:04:19.593 INFO:teuthology.orchestra.run.smithi078.stdout: "cluster_fsid": "e2ca1a0e-2277-11ee-9b33-001a4aab830c", 2023-07-14T19:04:19.593 INFO:teuthology.orchestra.run.smithi078.stdout: "whoami": 0, 2023-07-14T19:04:19.594 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 24503, 2023-07-14T19:04:19.594 INFO:teuthology.orchestra.run.smithi078.stdout: "want_state": "up:active", 2023-07-14T19:04:19.594 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "up:active", 2023-07-14T19:04:19.594 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs", 2023-07-14T19:04:19.594 INFO:teuthology.orchestra.run.smithi078.stdout: "rank_uptime": 108.64809099199999, 2023-07-14T19:04:19.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mdsmap_epoch": 23, 2023-07-14T19:04:19.595 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch": 87, 2023-07-14T19:04:19.595 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap_epoch_barrier": 85, 2023-07-14T19:04:19.595 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 114.65984664 2023-07-14T19:04:19.595 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:04:19.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 -- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a1150 msgr2=0x7fc3a40a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a1150 0x7fc3a40a3600 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc3b0005fd0 tx=0x7fc3b0005f00 comp rx=0 tx=0).stop 2023-07-14T19:04:19.596 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 -- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 msgr2=0x7fc3bc072a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:19.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 0x7fc3bc072a30 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac005f50 tx=0x7fc3ac0047a0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 -- 172.21.15.78:0/2460687582 shutdown_connections 2023-07-14T19:04:19.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc3a40a1150 0x7fc3a40a3600 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc3bc1017c0 0x7fc3bc072f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc3bc0ffc40 0x7fc3bc0743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 --2- 172.21.15.78:0/2460687582 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc3bc100e30 0x7fc3bc072a30 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:19.597 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 -- 172.21.15.78:0/2460687582 >> 172.21.15.78:0/2460687582 conn(0x7fc3bc0fb240 msgr2=0x7fc3bc0fd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:19.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 -- 172.21.15.78:0/2460687582 shutdown_connections 2023-07-14T19:04:19.598 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:19.577+0000 7fc3c245f700 1 -- 172.21.15.78:0/2460687582 wait complete. 2023-07-14T19:04:19.598 INFO:teuthology.run_tasks:Running task ceph-fuse... 2023-07-14T19:04:19.610 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2023-07-14T19:04:19.610 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2023-07-14T19:04:19.611 INFO:tasks.ceph_fuse:client.0 config is {'mount_subvol_num': 0} 2023-07-14T19:04:19.611 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-07-14T19:04:19.611 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T19:04:19.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:19.612 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-07-14T19:04:19.629 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2023-07-14T19:04:19.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:19.630 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2023-07-14T19:04:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:19 smithi102 ceph-mon[129280]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-14T19:04:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:19 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1874159743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:19.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:19 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1566172531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:19.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:19 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/4173388079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:19.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:19.709 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2023-07-14T19:04:19.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:19 smithi088 ceph-mon[128973]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 0 B/s wr, 5 op/s 2023-07-14T19:04:19.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:19 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1874159743' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:19.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:19 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1566172531' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:19.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:19 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/4173388079' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:19.806 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2023-07-14T19:04:19.807 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:81c9387d2e7576da065747ec9c16d1c7d69016ec shell --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c -- ceph fs ls 2023-07-14T19:04:20.357 INFO:teuthology.orchestra.run.smithi078.stderr:Inferring config /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/mon.a/config 2023-07-14T19:04:21.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 -- 172.21.15.78:0/529103106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 msgr2=0x7f3f6810e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:21.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 --2- 172.21.15.78:0/529103106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 0x7f3f6810e9f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000b0a0 tx=0x7f3f6000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 -- 172.21.15.78:0/529103106 shutdown_connections 2023-07-14T19:04:21.534 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 --2- 172.21.15.78:0/529103106 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 0x7f3f6810e9f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 --2- 172.21.15.78:0/529103106 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f68072960 0x7f3f68072d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 --2- 172.21.15.78:0/529103106 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f68073270 0x7f3f6810c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 -- 172.21.15.78:0/529103106 >> 172.21.15.78:0/529103106 conn(0x7f3f68077cd0 msgr2=0x7f3f680780d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 -- 172.21.15.78:0/529103106 shutdown_connections 2023-07-14T19:04:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.530+0000 7f3f6e881700 1 -- 172.21.15.78:0/529103106 wait complete. 2023-07-14T19:04:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6e881700 1 Processor -- start 2023-07-14T19:04:21.535 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6e881700 1 -- start start 2023-07-14T19:04:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6e881700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f68072960 0x7f3f68198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6e881700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f68073270 0x7f3f68198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6e881700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 0x7f3f6819c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:21.538 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6e881700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6819cfc0 con 0x7f3f6810c610 2023-07-14T19:04:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6e881700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6819d130 con 0x7f3f68073270 2023-07-14T19:04:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6e881700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6819d2f0 con 0x7f3f68072960 2023-07-14T19:04:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6ce1e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 0x7f3f6819c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6ce1e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 0x7f3f6819c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36242/0 (socket says 172.21.15.78:36242) 2023-07-14T19:04:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f6ce1e700 1 -- 172.21.15.78:0/2374035041 learned_addr learned my addr 172.21.15.78:0/2374035041 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f67fff700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f68072960 0x7f3f68198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:21.539 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.531+0000 7f3f677fe700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f68073270 0x7f3f68198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:21.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f6ce1e700 1 -- 172.21.15.78:0/2374035041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f68072960 msgr2=0x7f3f68198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:21.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f6ce1e700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f68072960 0x7f3f68198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f6ce1e700 1 -- 172.21.15.78:0/2374035041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f68073270 msgr2=0x7f3f68198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:21.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f6ce1e700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f68073270 0x7f3f68198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f6ce1e700 1 -- 172.21.15.78:0/2374035041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f60009d00 con 0x7f3f6810c610 2023-07-14T19:04:21.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f677fe700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f68073270 0x7f3f68198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:21.540 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f6ce1e700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 0x7f3f6819c9b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000b830 tx=0x7f3f600094e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:21.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f657fa700 1 -- 172.21.15.78:0/2374035041 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6000e040 con 0x7f3f6810c610 2023-07-14T19:04:21.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6819d570 con 0x7f3f6810c610 2023-07-14T19:04:21.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f657fa700 1 -- 172.21.15.78:0/2374035041 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3f60003e90 con 0x7f3f6810c610 2023-07-14T19:04:21.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f657fa700 1 -- 172.21.15.78:0/2374035041 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f6001d480 con 0x7f3f6810c610 2023-07-14T19:04:21.541 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.532+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f681a4e30 con 0x7f3f6810c610 2023-07-14T19:04:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.533+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f68106f30 con 0x7f3f6810c610 2023-07-14T19:04:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.534+0000 7f3f657fa700 1 -- 172.21.15.78:0/2374035041 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f3f60003a00 con 0x7f3f6810c610 2023-07-14T19:04:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.534+0000 7f3f657fa700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3f540a1150 0x7f3f540a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.535+0000 7f3f67fff700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3f540a1150 0x7f3f540a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.535+0000 7f3f657fa700 1 -- 172.21.15.78:0/2374035041 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f3f600ce160 con 0x7f3f6810c610 2023-07-14T19:04:21.542 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.535+0000 7f3f67fff700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3f540a1150 0x7f3f540a3600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f58005950 tx=0x7f3f5800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:21.543 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.539+0000 7f3f657fa700 1 -- 172.21.15.78:0/2374035041 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3f60097630 con 0x7f3f6810c610 2023-07-14T19:04:21.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:21 smithi078 ceph-mon[127425]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:04:21.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:21 smithi102 ceph-mon[129280]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:04:21.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:21 smithi088 ceph-mon[128973]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2023-07-14T19:04:21.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.769+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f3f68106910 con 0x7f3f6810c610 2023-07-14T19:04:21.773 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.770+0000 7f3f657fa700 1 -- 172.21.15.78:0/2374035041 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) v1 ==== 53+0+88 (secure 0 0 0) 0x7f3f6009ac50 con 0x7f3f6810c610 2023-07-14T19:04:21.773 INFO:teuthology.orchestra.run.smithi078.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2023-07-14T19:04:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3f540a1150 msgr2=0x7f3f540a3600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3f540a1150 0x7f3f540a3600 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3f58005950 tx=0x7f3f5800b410 comp rx=0 tx=0).stop 2023-07-14T19:04:21.775 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 msgr2=0x7f3f6819c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 0x7f3f6819c9b0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3f6000b830 tx=0x7f3f600094e0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 shutdown_connections 2023-07-14T19:04:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3f540a1150 0x7f3f540a3600 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3f6810c610 0x7f3f6819c9b0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f68072960 0x7f3f68198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 --2- 172.21.15.78:0/2374035041 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3f68073270 0x7f3f68198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:21.776 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 >> 172.21.15.78:0/2374035041 conn(0x7f3f68077cd0 msgr2=0x7f3f68104070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 shutdown_connections 2023-07-14T19:04:21.777 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:21.772+0000 7f3f6e881700 1 -- 172.21.15.78:0/2374035041 wait complete. 2023-07-14T19:04:22.203 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-07-14T19:04:22.203 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-07-14T19:04:22.203 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi078.front.sepia.ceph.com 2023-07-14T19:04:22.204 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-07-14T19:04:22.204 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-07-14T19:04:22.204 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-07-14T19:04:22.204 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-07-14T19:04:22.205 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-07-14T19:04:22.205 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-07-14T19:04:22.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.205 DEBUG:teuthology.orchestra.run.smithi078:> ip addr 2023-07-14T19:04:22.223 INFO:teuthology.orchestra.run.smithi078.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-07-14T19:04:22.224 INFO:teuthology.orchestra.run.smithi078.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-07-14T19:04:22.224 INFO:teuthology.orchestra.run.smithi078.stdout: inet 127.0.0.1/8 scope host lo 2023-07-14T19:04:22.224 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:04:22.224 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 ::1/128 scope host 2023-07-14T19:04:22.225 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:04:22.225 INFO:teuthology.orchestra.run.smithi078.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:04:22.225 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:22 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:04:22.225 INFO:teuthology.orchestra.run.smithi078.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:04:22.225 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:d8:a1:23 brd ff:ff:ff:ff:ff:ff 2023-07-14T19:04:22.226 INFO:teuthology.orchestra.run.smithi078.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-07-14T19:04:22.226 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dc brd ff:ff:ff:ff:ff:ff 2023-07-14T19:04:22.226 INFO:teuthology.orchestra.run.smithi078.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-07-14T19:04:22.226 INFO:teuthology.orchestra.run.smithi078.stdout: link/ether 0c:c4:7a:8f:d4:dd brd ff:ff:ff:ff:ff:ff 2023-07-14T19:04:22.226 INFO:teuthology.orchestra.run.smithi078.stdout: inet 172.21.15.78/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-07-14T19:04:22.226 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft 41403sec preferred_lft 41403sec 2023-07-14T19:04:22.226 INFO:teuthology.orchestra.run.smithi078.stdout: inet6 fe80::8e6d:82b2:36a8:5b8d/64 scope link noprefixroute 2023-07-14T19:04:22.227 INFO:teuthology.orchestra.run.smithi078.stdout: valid_lft forever preferred_lft forever 2023-07-14T19:04:22.227 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-07-14T19:04:22.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.227 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:04:22.228 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:04:22.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add name ceph-brx type bridge 2023-07-14T19:04:22.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr flush dev ceph-brx 2023-07-14T19:04:22.228 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set ceph-brx up 2023-07-14T19:04:22.229 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-07-14T19:04:22.229 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:04:22.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.343 DEBUG:teuthology.orchestra.run.smithi078:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-07-14T19:04:22.370 INFO:teuthology.orchestra.run.smithi078.stdout:1 2023-07-14T19:04:22.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.372 DEBUG:teuthology.orchestra.run.smithi078:> route 2023-07-14T19:04:22.430 INFO:teuthology.orchestra.run.smithi078.stdout:Kernel IP routing table 2023-07-14T19:04:22.430 INFO:teuthology.orchestra.run.smithi078.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-07-14T19:04:22.430 INFO:teuthology.orchestra.run.smithi078.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-07-14T19:04:22.430 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-07-14T19:04:22.430 INFO:teuthology.orchestra.run.smithi078.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-07-14T19:04:22.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.431 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:04:22.431 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:04:22.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-07-14T19:04:22.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-07-14T19:04:22.432 DEBUG:teuthology.orchestra.run.smithi078:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-07-14T19:04:22.432 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:04:22.508 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:22 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2374035041' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:04:22.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.536 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-07-14T19:04:22.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.595 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list-id 2023-07-14T19:04:22.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:22 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2374035041' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:04:22.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.651 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:04:22.651 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:04:22.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T19:04:22.652 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-07-14T19:04:22.652 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:04:22.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:22 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2374035041' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2023-07-14T19:04:22.745 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-07-14T19:04:22.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.746 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:04:22.746 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:04:22.747 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-07-14T19:04:22.747 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-07-14T19:04:22.747 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-07-14T19:04:22.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-07-14T19:04:22.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-07-14T19:04:22.748 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:04:22.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:22.953 DEBUG:teuthology.orchestra.run.smithi078:> (cd / && exec bash -c ' 2023-07-14T19:04:22.953 DEBUG:teuthology.orchestra.run.smithi078:> set -e 2023-07-14T19:04:22.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set brx.0 up 2023-07-14T19:04:22.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link set dev brx.0 master ceph-brx 2023-07-14T19:04:22.954 DEBUG:teuthology.orchestra.run.smithi078:> ') 2023-07-14T19:04:23.079 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {'mount_subvol_num': 0} 2023-07-14T19:04:23.079 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:04:23.079 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-07-14T19:04:23.099 INFO:teuthology.orchestra.run.smithi078.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-07-14T19:04:23.100 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-07-14T19:04:23.100 DEBUG:teuthology.orchestra.run.smithi078:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-07-14T19:04:23.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2023-07-14T19:04:23.223 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T19:04:23.224 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T19:04:23.290 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T19:04:23.291 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T19:04:23.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:23.292 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2023-07-14T19:04:23.348 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2023-07-14T19:04:23.349 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0 --client_mountpoint=/volumes/_nogroup/sv_0/9431b327-2d1c-4178-bfca-51125bd3288c --client_fs=cephfs) 2023-07-14T19:04:23.390 DEBUG:teuthology.orchestra.run.smithi078:> sudo modprobe fuse 2023-07-14T19:04:23.411 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:23 smithi078 ceph-mon[127425]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:04:23.417 INFO:teuthology.orchestra.run:Running command with timeout 30 2023-07-14T19:04:23.417 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2023-07-14T19:04:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi078.stderr:2023-07-14T19:04:23.475+0000 7fe6aeff75c0 -1 init, newargv = 0x556c34637fd0 newargc=15 2023-07-14T19:04:23.481 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi078.stderr:ceph-fuse[178281]: starting ceph client 2023-07-14T19:04:23.486 INFO:teuthology.orchestra.run.smithi078.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2023-07-14T19:04:23.486 DEBUG:teuthology.orchestra.run:got remote process result: 32 2023-07-14T19:04:23.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:23.488 DEBUG:teuthology.orchestra.run.smithi078:> ls /sys/fs/fuse/connections 2023-07-14T19:04:23.495 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi078.stderr:ceph-fuse[178281]: starting fuse 2023-07-14T19:04:23.514 INFO:teuthology.orchestra.run.smithi078.stdout:124 2023-07-14T19:04:23.514 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [124] 2023-07-14T19:04:23.515 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2023-07-14T19:04:23.515 DEBUG:teuthology.orchestra.run.smithi078:> import glob 2023-07-14T19:04:23.516 DEBUG:teuthology.orchestra.run.smithi078:> import re 2023-07-14T19:04:23.516 DEBUG:teuthology.orchestra.run.smithi078:> import os 2023-07-14T19:04:23.516 DEBUG:teuthology.orchestra.run.smithi078:> import subprocess 2023-07-14T19:04:23.517 DEBUG:teuthology.orchestra.run.smithi078:> 2023-07-14T19:04:23.517 DEBUG:teuthology.orchestra.run.smithi078:> def _find_admin_socket(client_name): 2023-07-14T19:04:23.517 DEBUG:teuthology.orchestra.run.smithi078:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2023-07-14T19:04:23.517 DEBUG:teuthology.orchestra.run.smithi078:> files = glob.glob(asok_path) 2023-07-14T19:04:23.518 DEBUG:teuthology.orchestra.run.smithi078:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2023-07-14T19:04:23.518 DEBUG:teuthology.orchestra.run.smithi078:> 2023-07-14T19:04:23.518 DEBUG:teuthology.orchestra.run.smithi078:> # Given a non-glob path, it better be there 2023-07-14T19:04:23.518 DEBUG:teuthology.orchestra.run.smithi078:> if "*" not in asok_path: 2023-07-14T19:04:23.519 DEBUG:teuthology.orchestra.run.smithi078:> assert(len(files) == 1) 2023-07-14T19:04:23.519 DEBUG:teuthology.orchestra.run.smithi078:> return files[0] 2023-07-14T19:04:23.519 DEBUG:teuthology.orchestra.run.smithi078:> 2023-07-14T19:04:23.519 DEBUG:teuthology.orchestra.run.smithi078:> for f in files: 2023-07-14T19:04:23.520 DEBUG:teuthology.orchestra.run.smithi078:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2023-07-14T19:04:23.520 DEBUG:teuthology.orchestra.run.smithi078:> if os.path.exists("/proc/{0}".format(pid)): 2023-07-14T19:04:23.520 DEBUG:teuthology.orchestra.run.smithi078:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2023-07-14T19:04:23.521 DEBUG:teuthology.orchestra.run.smithi078:> contents = proc_f.read() 2023-07-14T19:04:23.521 DEBUG:teuthology.orchestra.run.smithi078:> if mountpoint in contents: 2023-07-14T19:04:23.521 DEBUG:teuthology.orchestra.run.smithi078:> return f 2023-07-14T19:04:23.521 DEBUG:teuthology.orchestra.run.smithi078:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2023-07-14T19:04:23.522 DEBUG:teuthology.orchestra.run.smithi078:> 2023-07-14T19:04:23.522 DEBUG:teuthology.orchestra.run.smithi078:> print(_find_admin_socket("client.0")) 2023-07-14T19:04:23.522 DEBUG:teuthology.orchestra.run.smithi078:> ' 2023-07-14T19:04:23.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:23 smithi102 ceph-mon[129280]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:04:23.637 INFO:teuthology.orchestra.run.smithi078.stdout:/var/run/ceph/ceph-client.0.178281.asok 2023-07-14T19:04:23.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:23 smithi088 ceph-mon[128973]: pgmap v313: 129 pgs: 129 active+clean; 637 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2023-07-14T19:04:23.820 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.178281.asok 2023-07-14T19:04:23.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:23.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.178281.asok status 2023-07-14T19:04:23.976 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2023-07-14T19:04:23.976 INFO:teuthology.orchestra.run.smithi078.stdout: "metadata": { 2023-07-14T19:04:23.976 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_sha1": "81c9387d2e7576da065747ec9c16d1c7d69016ec", 2023-07-14T19:04:23.977 INFO:teuthology.orchestra.run.smithi078.stdout: "ceph_version": "ceph version 18.0.0-4940-g81c9387d (81c9387d2e7576da065747ec9c16d1c7d69016ec) reef (dev)", 2023-07-14T19:04:23.977 INFO:teuthology.orchestra.run.smithi078.stdout: "entity_id": "0", 2023-07-14T19:04:23.977 INFO:teuthology.orchestra.run.smithi078.stdout: "hostname": "smithi078", 2023-07-14T19:04:23.977 INFO:teuthology.orchestra.run.smithi078.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2023-07-14T19:04:23.977 INFO:teuthology.orchestra.run.smithi078.stdout: "pid": "178281", 2023-07-14T19:04:23.977 INFO:teuthology.orchestra.run.smithi078.stdout: "root": "/volumes/_nogroup/sv_0/9431b327-2d1c-4178-bfca-51125bd3288c" 2023-07-14T19:04:23.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-14T19:04:23.977 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_count": 0, 2023-07-14T19:04:23.978 INFO:teuthology.orchestra.run.smithi078.stdout: "dentry_pinned_count": 0, 2023-07-14T19:04:23.978 INFO:teuthology.orchestra.run.smithi078.stdout: "id": 15456, 2023-07-14T19:04:23.978 INFO:teuthology.orchestra.run.smithi078.stdout: "inst": { 2023-07-14T19:04:23.978 INFO:teuthology.orchestra.run.smithi078.stdout: "name": { 2023-07-14T19:04:23.978 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "client", 2023-07-14T19:04:23.978 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 15456 2023-07-14T19:04:23.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-14T19:04:23.978 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2023-07-14T19:04:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2023-07-14T19:04:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2023-07-14T19:04:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1348112720 2023-07-14T19:04:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2023-07-14T19:04:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-14T19:04:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": { 2023-07-14T19:04:23.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2023-07-14T19:04:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "192.168.0.1:0", 2023-07-14T19:04:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 1348112720 2023-07-14T19:04:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2023-07-14T19:04:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "inst_str": "client.15456 192.168.0.1:0/1348112720", 2023-07-14T19:04:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "addr_str": "192.168.0.1:0/1348112720", 2023-07-14T19:04:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "inode_count": 5, 2023-07-14T19:04:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mds_epoch": 23, 2023-07-14T19:04:23.980 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch": 87, 2023-07-14T19:04:23.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osd_epoch_barrier": 0, 2023-07-14T19:04:23.981 INFO:teuthology.orchestra.run.smithi078.stdout: "blocklisted": false, 2023-07-14T19:04:23.981 INFO:teuthology.orchestra.run.smithi078.stdout: "fs_name": "cephfs" 2023-07-14T19:04:23.981 INFO:teuthology.orchestra.run.smithi078.stdout:} 2023-07-14T19:04:23.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:23.987 DEBUG:teuthology.orchestra.run.smithi078:> stat --file-system '--printf=%T 2023-07-14T19:04:23.987 DEBUG:teuthology.orchestra.run.smithi078:> ' -- /home/ubuntu/cephtest/mnt.0 2023-07-14T19:04:24.004 INFO:teuthology.orchestra.run.smithi078.stdout:fuseblk 2023-07-14T19:04:24.004 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2023-07-14T19:04:24.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:24.005 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-07-14T19:04:24.074 INFO:teuthology.run_tasks:Running task exec... 2023-07-14T19:04:24.084 INFO:teuthology.task.exec:Executing custom commands... 2023-07-14T19:04:24.085 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi078.front.sepia.ceph.com 2023-07-14T19:04:24.085 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2023-07-14T19:04:24.239 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.231+0000 7f134adec700 1 -- 172.21.15.78:0/1644189929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344068710 msgr2=0x7f1344102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:24.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.231+0000 7f134adec700 1 --2- 172.21.15.78:0/1644189929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344068710 0x7f1344102820 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f1338009990 tx=0x7f1338009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:24.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.231+0000 7f134adec700 1 -- 172.21.15.78:0/1644189929 shutdown_connections 2023-07-14T19:04:24.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.231+0000 7f134adec700 1 --2- 172.21.15.78:0/1644189929 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1344105810 0x7f1344107cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:24.240 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.231+0000 7f134adec700 1 --2- 172.21.15.78:0/1644189929 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344068710 0x7f1344102820 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:24.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.231+0000 7f134adec700 1 --2- 172.21.15.78:0/1644189929 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344102d60 0x7f1344105180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:24.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.231+0000 7f134adec700 1 -- 172.21.15.78:0/1644189929 >> 172.21.15.78:0/1644189929 conn(0x7f13440fa130 msgr2=0x7f13440fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:24.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 -- 172.21.15.78:0/1644189929 shutdown_connections 2023-07-14T19:04:24.241 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 -- 172.21.15.78:0/1644189929 wait complete. 2023-07-14T19:04:24.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 Processor -- start 2023-07-14T19:04:24.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 -- start start 2023-07-14T19:04:24.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1344068710 0x7f1344195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:24.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344102d60 0x7f1344195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:24.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344105810 0x7f1344196ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:24.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1344197340 con 0x7f1344102d60 2023-07-14T19:04:24.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13441974b0 con 0x7f1344068710 2023-07-14T19:04:24.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f134adec700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1344197670 con 0x7f1344105810 2023-07-14T19:04:24.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f1343fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344102d60 0x7f1344195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:24.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f1343fff700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344102d60 0x7f1344195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36270/0 (socket says 172.21.15.78:36270) 2023-07-14T19:04:24.244 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.232+0000 7f1343fff700 1 -- 172.21.15.78:0/3833382527 learned_addr learned my addr 172.21.15.78:0/3833382527 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1349389700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344105810 0x7f1344196ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1348b88700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1344068710 0x7f1344195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1343fff700 1 -- 172.21.15.78:0/3833382527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344105810 msgr2=0x7f1344196ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:24.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1343fff700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344105810 0x7f1344196ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:24.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1343fff700 1 -- 172.21.15.78:0/3833382527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1344068710 msgr2=0x7f1344195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:24.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1343fff700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1344068710 0x7f1344195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:24.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1343fff700 1 -- 172.21.15.78:0/3833382527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13380095f0 con 0x7f1344102d60 2023-07-14T19:04:24.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1349389700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344105810 0x7f1344196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:24.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1348b88700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1344068710 0x7f1344195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:24.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1343fff700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344102d60 0x7f1344195e60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f133000d950 tx=0x7f133000dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:24.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1341ffb700 1 -- 172.21.15.78:0/3833382527 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1330009b40 con 0x7f1344102d60 2023-07-14T19:04:24.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1341ffb700 1 -- 172.21.15.78:0/3833382527 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1330009ca0 con 0x7f1344102d60 2023-07-14T19:04:24.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1344197950 con 0x7f1344102d60 2023-07-14T19:04:24.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f1341ffb700 1 -- 172.21.15.78:0/3833382527 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133000f620 con 0x7f1344102d60 2023-07-14T19:04:24.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.233+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f134419e110 con 0x7f1344102d60 2023-07-14T19:04:24.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.234+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13440fbca0 con 0x7f1344102d60 2023-07-14T19:04:24.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.235+0000 7f1341ffb700 1 -- 172.21.15.78:0/3833382527 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7f1330010460 con 0x7f1344102d60 2023-07-14T19:04:24.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.236+0000 7f1341ffb700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f13340a11d0 0x7f13340a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:24.250 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.236+0000 7f1348b88700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f13340a11d0 0x7f13340a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:24.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.236+0000 7f1341ffb700 1 -- 172.21.15.78:0/3833382527 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7f13300ccf00 con 0x7f1344102d60 2023-07-14T19:04:24.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.236+0000 7f1348b88700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f13340a11d0 0x7f13340a3680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f13380095c0 tx=0x7f1338009510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:24.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.241+0000 7f1341ffb700 1 -- 172.21.15.78:0/3833382527 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f1330096340 con 0x7f1344102d60 2023-07-14T19:04:24.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:24.473+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f134404e420 con 0x7f1344102d60 2023-07-14T19:04:25.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.245+0000 7f1341ffb700 1 -- 172.21.15.78:0/3833382527 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v24) v1 ==== 118+0+0 (secure 0 0 0) 0x7f13300200b0 con 0x7f1344102d60 2023-07-14T19:04:25.251 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f13340a11d0 msgr2=0x7f13340a3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:25.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f13340a11d0 0x7f13340a3680 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f13380095c0 tx=0x7f1338009510 comp rx=0 tx=0).stop 2023-07-14T19:04:25.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344102d60 msgr2=0x7f1344195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:25.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344102d60 0x7f1344195e60 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f133000d950 tx=0x7f133000dd10 comp rx=0 tx=0).stop 2023-07-14T19:04:25.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 shutdown_connections 2023-07-14T19:04:25.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f13340a11d0 0x7f13340a3680 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1344068710 0x7f1344195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.252 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1344105810 0x7f1344196ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 --2- 172.21.15.78:0/3833382527 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1344102d60 0x7f1344195e60 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 >> 172.21.15.78:0/3833382527 conn(0x7f13440fa130 msgr2=0x7f13441010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:25.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 shutdown_connections 2023-07-14T19:04:25.253 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.248+0000 7f134adec700 1 -- 172.21.15.78:0/3833382527 wait complete. 2023-07-14T19:04:25.265 DEBUG:teuthology.orchestra.run.smithi078:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup distributed 1' 2023-07-14T19:04:25.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 -- 172.21.15.78:0/1285417309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1800ffd60 msgr2=0x7fc1801001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:25.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 --2- 172.21.15.78:0/1285417309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1800ffd60 0x7fc1801001d0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fc17c009990 tx=0x7fc17c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 -- 172.21.15.78:0/1285417309 shutdown_connections 2023-07-14T19:04:25.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 --2- 172.21.15.78:0/1285417309 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1800ff050 0x7fc1800ff460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 --2- 172.21.15.78:0/1285417309 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc180100710 0x7fc180106fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 --2- 172.21.15.78:0/1285417309 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1800ffd60 0x7fc1801001d0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 -- 172.21.15.78:0/1285417309 >> 172.21.15.78:0/1285417309 conn(0x7fc1800f98b0 msgr2=0x7fc1800fbd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:25.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 -- 172.21.15.78:0/1285417309 shutdown_connections 2023-07-14T19:04:25.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.387+0000 7fc1881e0700 1 -- 172.21.15.78:0/1285417309 wait complete. 2023-07-14T19:04:25.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc1881e0700 1 Processor -- start 2023-07-14T19:04:25.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc1881e0700 1 -- start start 2023-07-14T19:04:25.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc1881e0700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1800ff050 0x7fc180101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:25.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc1881e0700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1800ffd60 0x7fc180101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:25.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc1881e0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc180100710 0x7fc180101bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:25.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc1881e0700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18019f970 con 0x7fc1800ff050 2023-07-14T19:04:25.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc1881e0700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18019fab0 con 0x7fc1800ffd60 2023-07-14T19:04:25.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc1881e0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc18019fbf0 con 0x7fc180100710 2023-07-14T19:04:25.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1800ff050 0x7fc180101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:25.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1800ff050 0x7fc180101140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36282/0 (socket says 172.21.15.78:36282) 2023-07-14T19:04:25.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 -- 172.21.15.78:0/2981535483 learned_addr learned my addr 172.21.15.78:0/2981535483 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:25.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc18577b700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1800ffd60 0x7fc180101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:25.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc18677d700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc180100710 0x7fc180101bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:25.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 -- 172.21.15.78:0/2981535483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc180100710 msgr2=0x7fc180101bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:25.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc180100710 0x7fc180101bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 -- 172.21.15.78:0/2981535483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1800ffd60 msgr2=0x7fc180101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:25.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1800ffd60 0x7fc180101680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:25.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 -- 172.21.15.78:0/2981535483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c0095f0 con 0x7fc1800ff050 2023-07-14T19:04:25.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc18577b700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fc1800ffd60 0x7fc180101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:25.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.388+0000 7fc185f7c700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fc1800ff050 0x7fc180101140 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc17000b500 tx=0x7fc17000b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:25.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.389+0000 7fc176ffd700 1 -- 172.21.15.78:0/2981535483 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170011830 con 0x7fc1800ff050 2023-07-14T19:04:25.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.389+0000 7fc176ffd700 1 -- 172.21.15.78:0/2981535483 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fc170011e20 con 0x7fc1800ff050 2023-07-14T19:04:25.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.389+0000 7fc1881e0700 1 -- 172.21.15.78:0/2981535483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc18019fe80 con 0x7fc1800ff050 2023-07-14T19:04:25.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.389+0000 7fc176ffd700 1 -- 172.21.15.78:0/2981535483 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc170010dd0 con 0x7fc1800ff050 2023-07-14T19:04:25.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.389+0000 7fc1881e0700 1 -- 172.21.15.78:0/2981535483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1801a0320 con 0x7fc1800ff050 2023-07-14T19:04:25.398 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.389+0000 7fc1881e0700 1 -- 172.21.15.78:0/2981535483 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc18004e490 con 0x7fc1800ff050 2023-07-14T19:04:25.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.391+0000 7fc176ffd700 1 -- 172.21.15.78:0/2981535483 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 18) v1 ==== 135715+0+0 (secure 0 0 0) 0x7fc170011990 con 0x7fc1800ff050 2023-07-14T19:04:25.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.392+0000 7fc176ffd700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc16c0a1150 0x7fc16c0a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:25.399 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.392+0000 7fc176ffd700 1 -- 172.21.15.78:0/2981535483 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 9020+0+0 (secure 0 0 0) 0x7fc1700ccb20 con 0x7fc1800ff050 2023-07-14T19:04:25.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.392+0000 7fc18577b700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc16c0a1150 0x7fc16c0a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:25.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.392+0000 7fc18577b700 1 --2- 172.21.15.78:0/2981535483 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fc16c0a1150 0x7fc16c0a3600 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fc17c006010 tx=0x7fc17c009530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:25.400 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.395+0000 7fc176ffd700 1 -- 172.21.15.78:0/2981535483 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fc170095f60 con 0x7fc1800ff050 2023-07-14T19:04:25.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:25 smithi078 ceph-mon[127425]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:04:25.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:25 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3833382527' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:04:25.621 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.617+0000 7fc1881e0700 1 -- 172.21.15.78:0/2981535483 --> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "distributed", "pin_setting": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc1800fe2d0 con 0x7fc16c0a1150 2023-07-14T19:04:25.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:25 smithi102 ceph-mon[129280]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:04:25.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:25 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3833382527' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:04:25.631 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:25.627+0000 7fc176ffd700 1 -- 172.21.15.78:0/2981535483 <== mgr.14152 v2:172.21.15.78:6800/2266091126 1 ==== mgr_command_reply(tid 0: -1 Operation not permitted for group '_nogroup' as it is an internal group.) v1 ==== 80+0+0 (secure 0 0 0) 0x7fc1800fe2d0 con 0x7fc16c0a1150 2023-07-14T19:04:25.633 INFO:teuthology.orchestra.run.smithi078.stderr:Error EPERM: Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T19:04:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:04:25.637 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi078 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup distributed 1'" 2023-07-14T19:04:25.742 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=12a9a4a36cf644a8a96b101643743fc0 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/run_tasks.py", line 84, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_aa6a80e6cc655fff48452a7eca02070e61d074fc/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi078 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs _nogroup distributed 1'" 2023-07-14T19:04:25.745 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2023-07-14T19:04:25.757 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2023-07-14T19:04:25.758 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T19:04:25.758 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:04:30.767 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:04:25.621+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:04:30.768 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:04:25.621+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:04:30.768 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:04:25.621+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:04:30.768 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:04:25.621+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:04:30.769 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:04:25.621+0000 7ff2bf03b700 -1 client.0 error registering admin socket command: (17) File exists 2023-07-14T19:04:30.769 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:04:25.627+0000 7ff2bf03b700 -1 mgr.server reply reply (1) Operation not permitted Operation not permitted for group '_nogroup' as it is an internal group. 2023-07-14T19:04:30.770 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:26 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3833382527' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:04:30.770 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:26 smithi102 ceph-mon[129280]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:04:30.770 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:27 smithi102 ceph-mon[129280]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "distributed", "pin_setting": "1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:30.771 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:27 smithi102 ceph-mon[129280]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-14T19:04:30.771 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:28 smithi102 ceph-mon[129280]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T19:04:30.771 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:29 smithi102 ceph-mon[129280]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-14T19:04:30.771 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2023-07-14T19:04:30.771 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2023-07-14T19:04:30.772 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2023-07-14T19:04:30.772 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:30 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2023-07-14T19:04:30.772 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:25 smithi088 ceph-mon[128973]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 255 B/s wr, 6 op/s 2023-07-14T19:04:30.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:25 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3833382527' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2023-07-14T19:04:30.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:26 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3833382527' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:04:30.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:26 smithi088 ceph-mon[128973]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:04:30.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:27 smithi088 ceph-mon[128973]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "distributed", "pin_setting": "1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:30.773 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:27 smithi088 ceph-mon[128973]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-14T19:04:30.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:28 smithi088 ceph-mon[128973]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T19:04:30.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:29 smithi088 ceph-mon[128973]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-14T19:04:30.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2023-07-14T19:04:30.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2023-07-14T19:04:30.774 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2023-07-14T19:04:30.775 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:30 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2023-07-14T19:04:30.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:26 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3833382527' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2023-07-14T19:04:30.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:26 smithi078 ceph-mon[127425]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2023-07-14T19:04:30.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:27 smithi078 ceph-mon[127425]: from='client.15468 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "_nogroup", "pin_type": "distributed", "pin_setting": "1", "target": ["mon-mgr", ""]}]: dispatch 2023-07-14T19:04:30.775 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:27 smithi078 ceph-mon[127425]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 341 B/s wr, 5 op/s 2023-07-14T19:04:30.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:28 smithi078 ceph-mon[127425]: mgrmap e19: x(active, since 7m), standbys: z, y 2023-07-14T19:04:30.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:29 smithi078 ceph-mon[127425]: pgmap v316: 129 pgs: 129 active+clean; 639 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 682 B/s wr, 6 op/s 2023-07-14T19:04:30.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2023-07-14T19:04:30.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2023-07-14T19:04:30.776 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2023-07-14T19:04:30.777 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:30 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]: dispatch 2023-07-14T19:04:30.788 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T19:04:30.788 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:04:30.851 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:04:31.026 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 -- 172.21.15.78:0/944352085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780ff710 msgr2=0x7f0978106fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 --2- 172.21.15.78:0/944352085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780ff710 0x7f0978106fe0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f097000b0a0 tx=0x7f097000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 -- 172.21.15.78:0/944352085 shutdown_connections 2023-07-14T19:04:31.027 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 --2- 172.21.15.78:0/944352085 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780ff710 0x7f0978106fe0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 --2- 172.21.15.78:0/944352085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09781000e0 0x7f09781004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 --2- 172.21.15.78:0/944352085 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780fed80 0x7f09780ff1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 -- 172.21.15.78:0/944352085 >> 172.21.15.78:0/944352085 conn(0x7f09780f98a0 msgr2=0x7f09780fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:31.028 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 -- 172.21.15.78:0/944352085 shutdown_connections 2023-07-14T19:04:31.029 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.023+0000 7f097f90e700 1 -- 172.21.15.78:0/944352085 wait complete. 2023-07-14T19:04:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097f90e700 1 Processor -- start 2023-07-14T19:04:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097f90e700 1 -- start start 2023-07-14T19:04:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097f90e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780fed80 0x7f0978102b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097f90e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780ff710 0x7f0978101170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.031 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097f90e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09781000e0 0x7f09781016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097f90e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0978101e50 con 0x7f09780fed80 2023-07-14T19:04:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097f90e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0978101fc0 con 0x7f09780ff710 2023-07-14T19:04:31.032 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097f90e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f097819f8c0 con 0x7f09781000e0 2023-07-14T19:04:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097d6aa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780fed80 0x7f0978102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097d6aa700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780fed80 0x7f0978102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36310/0 (socket says 172.21.15.78:36310) 2023-07-14T19:04:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097d6aa700 1 -- 172.21.15.78:0/349479672 learned_addr learned my addr 172.21.15.78:0/349479672 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:31.033 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097cea9700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780ff710 0x7f0978101170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097d6aa700 1 -- 172.21.15.78:0/349479672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09781000e0 msgr2=0x7f09781016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097deab700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09781000e0 0x7f09781016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097d6aa700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09781000e0 0x7f09781016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097d6aa700 1 -- 172.21.15.78:0/349479672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780ff710 msgr2=0x7f0978101170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.034 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097d6aa700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780ff710 0x7f0978101170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097d6aa700 1 -- 172.21.15.78:0/349479672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0970009d00 con 0x7f09780fed80 2023-07-14T19:04:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.024+0000 7f097deab700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09781000e0 0x7f09781016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.025+0000 7f097d6aa700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780fed80 0x7f0978102b30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f096800c990 tx=0x7f096800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.025+0000 7f096e7fc700 1 -- 172.21.15.78:0/349479672 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968007b00 con 0x7f09780fed80 2023-07-14T19:04:31.035 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.025+0000 7f096e7fc700 1 -- 172.21.15.78:0/349479672 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0968007c60 con 0x7f09780fed80 2023-07-14T19:04:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.025+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f097819fba0 con 0x7f09780fed80 2023-07-14T19:04:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.025+0000 7f096e7fc700 1 -- 172.21.15.78:0/349479672 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0968018660 con 0x7f09780fed80 2023-07-14T19:04:31.036 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.025+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09781a0100 con 0x7f09780fed80 2023-07-14T19:04:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.026+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f097810a190 con 0x7f09780fed80 2023-07-14T19:04:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.027+0000 7f096e7fc700 1 -- 172.21.15.78:0/349479672 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f096800f450 con 0x7f09780fed80 2023-07-14T19:04:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.027+0000 7f096e7fc700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f09640a12a0 0x7f09640a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.028+0000 7f097cea9700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f09640a12a0 0x7f09640a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.037 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.028+0000 7f096e7fc700 1 -- 172.21.15.78:0/349479672 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f09680cce40 con 0x7f09780fed80 2023-07-14T19:04:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.028+0000 7f097cea9700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f09640a12a0 0x7f09640a3750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0974005950 tx=0x7f097400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:31.038 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.031+0000 7f096e7fc700 1 -- 172.21.15.78:0/349479672 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f09680961c0 con 0x7f09780fed80 2023-07-14T19:04:31.223 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.219+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f097810a3b0 con 0x7f09780fed80 2023-07-14T19:04:31.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.221+0000 7f096e7fc700 1 -- 172.21.15.78:0/349479672 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+24640 (secure 0 0 0) 0x7f096800fe60 con 0x7f09780fed80 2023-07-14T19:04:31.225 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:31.225 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":88,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:04:30.274721+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:04:31.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f09640a12a0 msgr2=0x7f09640a3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f09640a12a0 0x7f09640a3750 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f0974005950 tx=0x7f097400b410 comp rx=0 tx=0).stop 2023-07-14T19:04:31.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780fed80 msgr2=0x7f0978102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780fed80 0x7f0978102b30 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f096800c990 tx=0x7f096800cca0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 shutdown_connections 2023-07-14T19:04:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f09640a12a0 0x7f09640a3750 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09781000e0 0x7f09781016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f09780fed80 0x7f0978102b30 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.228 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 --2- 172.21.15.78:0/349479672 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f09780ff710 0x7f0978101170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 >> 172.21.15.78:0/349479672 conn(0x7f09780f98a0 msgr2=0x7f09780fa030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 shutdown_connections 2023-07-14T19:04:31.229 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.223+0000 7f097f90e700 1 -- 172.21.15.78:0/349479672 wait complete. 2023-07-14T19:04:31.239 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:02:28.269050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:02:29.547771+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T19:02:32.797605+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T19:04:31.240 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:04:31.369 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 -- 172.21.15.78:0/2530949937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 msgr2=0x7f77b4101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 --2- 172.21.15.78:0/2530949937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 0x7f77b4101300 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f77b0009990 tx=0x7f77b0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 -- 172.21.15.78:0/2530949937 shutdown_connections 2023-07-14T19:04:31.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 --2- 172.21.15.78:0/2530949937 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77b40ffc00 0x7f77b4100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.370 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 --2- 172.21.15.78:0/2530949937 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b4101840 0x7f77b41080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 --2- 172.21.15.78:0/2530949937 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 0x7f77b4101300 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 -- 172.21.15.78:0/2530949937 >> 172.21.15.78:0/2530949937 conn(0x7f77b40fb220 msgr2=0x7f77b40fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 -- 172.21.15.78:0/2530949937 shutdown_connections 2023-07-14T19:04:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 -- 172.21.15.78:0/2530949937 wait complete. 2023-07-14T19:04:31.371 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 Processor -- start 2023-07-14T19:04:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.365+0000 7f77bbc32700 1 -- start start 2023-07-14T19:04:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77bbc32700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b40ffc00 0x7f77b4197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77bbc32700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 0x7f77b4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77bbc32700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77b4101840 0x7f77b419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77bbc32700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b419f650 con 0x7f77b4100e90 2023-07-14T19:04:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77bbc32700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b419f790 con 0x7f77b4101840 2023-07-14T19:04:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77bbc32700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b419f920 con 0x7f77b40ffc00 2023-07-14T19:04:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 0x7f77b4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 0x7f77b4198010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36326/0 (socket says 172.21.15.78:36326) 2023-07-14T19:04:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 -- 172.21.15.78:0/3353384779 learned_addr learned my addr 172.21.15.78:0/3353384779 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b99ce700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b40ffc00 0x7f77b4197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77ba1cf700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77b4101840 0x7f77b419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 -- 172.21.15.78:0/3353384779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b40ffc00 msgr2=0x7f77b4197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b40ffc00 0x7f77b4197ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 -- 172.21.15.78:0/3353384779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77b4101840 msgr2=0x7f77b419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77b4101840 0x7f77b419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 -- 172.21.15.78:0/3353384779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b00095f0 con 0x7f77b4100e90 2023-07-14T19:04:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77ba1cf700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77b4101840 0x7f77b419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77b91cd700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 0x7f77b4198010 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f77b000c010 tx=0x7f77b0003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:31.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77aaffd700 1 -- 172.21.15.78:0/3353384779 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b001d070 con 0x7f77b4100e90 2023-07-14T19:04:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77aaffd700 1 -- 172.21.15.78:0/3353384779 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f77b00041d0 con 0x7f77b4100e90 2023-07-14T19:04:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77b419fba0 con 0x7f77b4100e90 2023-07-14T19:04:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77aaffd700 1 -- 172.21.15.78:0/3353384779 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77b000f460 con 0x7f77b4100e90 2023-07-14T19:04:31.376 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.366+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77b419ffc0 con 0x7f77b4100e90 2023-07-14T19:04:31.377 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.367+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b410b2d0 con 0x7f77b4100e90 2023-07-14T19:04:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.369+0000 7f77aaffd700 1 -- 172.21.15.78:0/3353384779 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f77b0022670 con 0x7f77b4100e90 2023-07-14T19:04:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.369+0000 7f77aaffd700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77a00a11e0 0x7f77a00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.369+0000 7f77b99ce700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77a00a11e0 0x7f77a00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.369+0000 7f77aaffd700 1 -- 172.21.15.78:0/3353384779 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f77b00ce970 con 0x7f77b4100e90 2023-07-14T19:04:31.378 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.370+0000 7f77b99ce700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77a00a11e0 0x7f77a00a3690 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f77a4005fd0 tx=0x7f77a400a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:31.379 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.373+0000 7f77aaffd700 1 -- 172.21.15.78:0/3353384779 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f77b0097c70 con 0x7f77b4100e90 2023-07-14T19:04:31.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:31 smithi078 ceph-mon[127425]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-07-14T19:04:31.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2023-07-14T19:04:31.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2023-07-14T19:04:31.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2023-07-14T19:04:31.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:31 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2023-07-14T19:04:31.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:31 smithi078 ceph-mon[127425]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T19:04:31.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:31 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/349479672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:31.565 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.562+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f77b41a05d0 con 0x7f77b4100e90 2023-07-14T19:04:31.566 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.563+0000 7f77aaffd700 1 -- 172.21.15.78:0/3353384779 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f77b009b290 con 0x7f77b4100e90 2023-07-14T19:04:31.566 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:04:31.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77a00a11e0 msgr2=0x7f77a00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.568 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77a00a11e0 0x7f77a00a3690 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f77a4005fd0 tx=0x7f77a400a560 comp rx=0 tx=0).stop 2023-07-14T19:04:31.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 msgr2=0x7f77b4198010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 0x7f77b4198010 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f77b000c010 tx=0x7f77b0003820 comp rx=0 tx=0).stop 2023-07-14T19:04:31.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 shutdown_connections 2023-07-14T19:04:31.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f77a00a11e0 0x7f77a00a3690 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.569 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f77b4101840 0x7f77b419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f77b40ffc00 0x7f77b4197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 --2- 172.21.15.78:0/3353384779 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f77b4100e90 0x7f77b4198010 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 >> 172.21.15.78:0/3353384779 conn(0x7f77b40fb220 msgr2=0x7f77b40fd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:31.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 shutdown_connections 2023-07-14T19:04:31.570 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.565+0000 7f77bbc32700 1 -- 172.21.15.78:0/3353384779 wait complete. 2023-07-14T19:04:31.581 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:04:31.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:31 smithi102 ceph-mon[129280]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-07-14T19:04:31.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2023-07-14T19:04:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2023-07-14T19:04:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2023-07-14T19:04:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:31 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2023-07-14T19:04:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:31 smithi102 ceph-mon[129280]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T19:04:31.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:31 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/349479672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:31.706 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 -- 172.21.15.78:0/1761837169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c0fed00 msgr2=0x7f3e9c0ff170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1761837169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c0fed00 0x7f3e9c0ff170 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009990 tx=0x7f3e98009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 -- 172.21.15.78:0/1761837169 shutdown_connections 2023-07-14T19:04:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1761837169 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e9c0ff6b0 0x7f3e9c106f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.707 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1761837169 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e9c1000b0 0x7f3e9c1004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1761837169 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c0fed00 0x7f3e9c0ff170 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 -- 172.21.15.78:0/1761837169 >> 172.21.15.78:0/1761837169 conn(0x7f3e9c0f98a0 msgr2=0x7f3e9c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:31.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 -- 172.21.15.78:0/1761837169 shutdown_connections 2023-07-14T19:04:31.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.701+0000 7f3ea32de700 1 -- 172.21.15.78:0/1761837169 wait complete. 2023-07-14T19:04:31.708 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea32de700 1 Processor -- start 2023-07-14T19:04:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea32de700 1 -- start start 2023-07-14T19:04:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea32de700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e9c0fed00 0x7f3e9c10ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea32de700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e9c0ff6b0 0x7f3e9c10ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea32de700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c1000b0 0x7f3e9c115fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.709 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea32de700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c1165a0 con 0x7f3e9c1000b0 2023-07-14T19:04:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea32de700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c116710 con 0x7f3e9c0fed00 2023-07-14T19:04:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea32de700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e9c1ac650 con 0x7f3e9c0ff6b0 2023-07-14T19:04:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea187b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c1000b0 0x7f3e9c115fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea187b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c1000b0 0x7f3e9c115fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36342/0 (socket says 172.21.15.78:36342) 2023-07-14T19:04:31.710 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea187b700 1 -- 172.21.15.78:0/1503035392 learned_addr learned my addr 172.21.15.78:0/1503035392 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea107a700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e9c0fed00 0x7f3e9c10ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea0879700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e9c0ff6b0 0x7f3e9c10ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea187b700 1 -- 172.21.15.78:0/1503035392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e9c0ff6b0 msgr2=0x7f3e9c10ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea187b700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e9c0ff6b0 0x7f3e9c10ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.711 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.702+0000 7f3ea187b700 1 -- 172.21.15.78:0/1503035392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e9c0fed00 msgr2=0x7f3e9c10ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3ea187b700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e9c0fed00 0x7f3e9c10ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3ea187b700 1 -- 172.21.15.78:0/1503035392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e980095f0 con 0x7f3e9c1000b0 2023-07-14T19:04:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3ea107a700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e9c0fed00 0x7f3e9c10ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3ea187b700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c1000b0 0x7f3e9c115fd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3e94007d70 tx=0x7f3e9400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:31.712 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3e927fc700 1 -- 172.21.15.78:0/1503035392 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e9400ad40 con 0x7f3e9c1000b0 2023-07-14T19:04:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3e927fc700 1 -- 172.21.15.78:0/1503035392 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f3e940128b0 con 0x7f3e9c1000b0 2023-07-14T19:04:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e9c1ac8b0 con 0x7f3e9c1000b0 2023-07-14T19:04:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3e927fc700 1 -- 172.21.15.78:0/1503035392 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e94011620 con 0x7f3e9c1000b0 2023-07-14T19:04:31.713 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.703+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e9c1acd90 con 0x7f3e9c1000b0 2023-07-14T19:04:31.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:31 smithi088 ceph-mon[128973]: pgmap v317: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-07-14T19:04:31.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2023-07-14T19:04:31.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2023-07-14T19:04:31.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2023-07-14T19:04:31.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:31 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.37", "id": [5, 6]}]': finished 2023-07-14T19:04:31.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:31 smithi088 ceph-mon[128973]: osdmap e88: 12 total, 12 up, 12 in 2023-07-14T19:04:31.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:31 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/349479672' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:31.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.704+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e9c105e00 con 0x7f3e9c1000b0 2023-07-14T19:04:31.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.705+0000 7f3e927fc700 1 -- 172.21.15.78:0/1503035392 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f3e94012a20 con 0x7f3e9c1000b0 2023-07-14T19:04:31.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.705+0000 7f3e927fc700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e880a11e0 0x7f3e880a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:31.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.706+0000 7f3ea107a700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e880a11e0 0x7f3e880a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:31.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.706+0000 7f3e927fc700 1 -- 172.21.15.78:0/1503035392 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f3e940cd560 con 0x7f3e9c1000b0 2023-07-14T19:04:31.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.706+0000 7f3ea107a700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e880a11e0 0x7f3e880a3690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c005fd0 tx=0x7f3e8c005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:31.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.709+0000 7f3e927fc700 1 -- 172.21.15.78:0/1503035392 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f3e9409a370 con 0x7f3e9c1000b0 2023-07-14T19:04:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.895+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3e9c02cce0 con 0x7f3e9c1000b0 2023-07-14T19:04:31.917 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.896+0000 7f3e927fc700 1 -- 172.21.15.78:0/1503035392 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3e94099f00 con 0x7f3e9c1000b0 2023-07-14T19:04:31.918 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:04:31.919 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e880a11e0 msgr2=0x7f3e880a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e880a11e0 0x7f3e880a3690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c005fd0 tx=0x7f3e8c005ee0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c1000b0 msgr2=0x7f3e9c115fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:31.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c1000b0 0x7f3e9c115fd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3e94007d70 tx=0x7f3e9400a340 comp rx=0 tx=0).stop 2023-07-14T19:04:31.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 shutdown_connections 2023-07-14T19:04:31.920 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f3e880a11e0 0x7f3e880a3690 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f3e9c0fed00 0x7f3e9c10ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f3e9c1000b0 0x7f3e9c115fd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 --2- 172.21.15.78:0/1503035392 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e9c0ff6b0 0x7f3e9c10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 >> 172.21.15.78:0/1503035392 conn(0x7f3e9c0f98a0 msgr2=0x7f3e9c102f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:31.921 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 shutdown_connections 2023-07-14T19:04:31.922 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:31.898+0000 7f3ea32de700 1 -- 172.21.15.78:0/1503035392 wait complete. 2023-07-14T19:04:31.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:04:32.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 -- 172.21.15.78:0/3471850496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 msgr2=0x7f1578100150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 --2- 172.21.15.78:0/3471850496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 0x7f1578100150 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f1568009990 tx=0x7f1568009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 -- 172.21.15.78:0/3471850496 shutdown_connections 2023-07-14T19:04:32.047 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 --2- 172.21.15.78:0/3471850496 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f15780feb10 0x7f15780fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 --2- 172.21.15.78:0/3471850496 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1578100690 0x7f1578106f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 --2- 172.21.15.78:0/3471850496 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 0x7f1578100150 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 -- 172.21.15.78:0/3471850496 >> 172.21.15.78:0/3471850496 conn(0x7f15780fa130 msgr2=0x7f15780fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:32.048 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 -- 172.21.15.78:0/3471850496 shutdown_connections 2023-07-14T19:04:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 -- 172.21.15.78:0/3471850496 wait complete. 2023-07-14T19:04:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 Processor -- start 2023-07-14T19:04:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.042+0000 7f157e337700 1 -- start start 2023-07-14T19:04:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f157e337700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15780feb10 0x7f1578068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.049 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f157e337700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 0x7f157819a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f157e337700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1578100690 0x7f157819cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f157e337700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819d410 con 0x7f15780ffd00 2023-07-14T19:04:32.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f157e337700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819d580 con 0x7f1578100690 2023-07-14T19:04:32.050 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f157e337700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f157819d740 con 0x7f15780feb10 2023-07-14T19:04:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 0x7f157819a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 0x7f157819a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36354/0 (socket says 172.21.15.78:36354) 2023-07-14T19:04:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 -- 172.21.15.78:0/3384009144 learned_addr learned my addr 172.21.15.78:0/3384009144 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:32.051 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f1577fff700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15780feb10 0x7f1578068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f157c8d4700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1578100690 0x7f157819cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 -- 172.21.15.78:0/3384009144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15780feb10 msgr2=0x7f1578068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.052 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15780feb10 0x7f1578068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 -- 172.21.15.78:0/3384009144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1578100690 msgr2=0x7f157819cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1578100690 0x7f157819cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 -- 172.21.15.78:0/3384009144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15680095f0 con 0x7f15780ffd00 2023-07-14T19:04:32.053 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15777fe700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 0x7f157819a5f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1568003a50 tx=0x7f1568009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15757fa700 1 -- 172.21.15.78:0/3384009144 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f156801d070 con 0x7f15780ffd00 2023-07-14T19:04:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15757fa700 1 -- 172.21.15.78:0/3384009144 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f1568005a00 con 0x7f15780ffd00 2023-07-14T19:04:32.054 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f157819d9c0 con 0x7f15780ffd00 2023-07-14T19:04:32.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.043+0000 7f15757fa700 1 -- 172.21.15.78:0/3384009144 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1568005020 con 0x7f15780ffd00 2023-07-14T19:04:32.055 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.044+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f157819de60 con 0x7f15780ffd00 2023-07-14T19:04:32.056 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.045+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f157810a190 con 0x7f15780ffd00 2023-07-14T19:04:32.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.046+0000 7f15757fa700 1 -- 172.21.15.78:0/3384009144 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f1568004800 con 0x7f15780ffd00 2023-07-14T19:04:32.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.046+0000 7f15757fa700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f15600a1260 0x7f15600a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.057 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.046+0000 7f15757fa700 1 -- 172.21.15.78:0/3384009144 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f15680ce2f0 con 0x7f15780ffd00 2023-07-14T19:04:32.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.046+0000 7f1577fff700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f15600a1260 0x7f15600a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.058 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.047+0000 7f1577fff700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f15600a1260 0x7f15600a3710 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f15780ffb60 tx=0x7f156c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:32.059 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.050+0000 7f15757fa700 1 -- 172.21.15.78:0/3384009144 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f15680975f0 con 0x7f15780ffd00 2023-07-14T19:04:32.242 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.239+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1578105ae0 con 0x7f15780ffd00 2023-07-14T19:04:32.243 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.239+0000 7f15757fa700 1 -- 172.21.15.78:0/3384009144 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f15680300a0 con 0x7f15780ffd00 2023-07-14T19:04:32.244 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-07-14T19:04:32.245 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f15600a1260 msgr2=0x7f15600a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f15600a1260 0x7f15600a3710 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f15780ffb60 tx=0x7f156c00b410 comp rx=0 tx=0).stop 2023-07-14T19:04:32.246 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 msgr2=0x7f157819a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 0x7f157819a5f0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f1568003a50 tx=0x7f1568009580 comp rx=0 tx=0).stop 2023-07-14T19:04:32.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 shutdown_connections 2023-07-14T19:04:32.247 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f15600a1260 0x7f15600a3710 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f1578100690 0x7f157819cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15780feb10 0x7f1578068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.248 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 --2- 172.21.15.78:0/3384009144 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f15780ffd00 0x7f157819a5f0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 >> 172.21.15.78:0/3384009144 conn(0x7f15780fa130 msgr2=0x7f15780fc480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:32.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 shutdown_connections 2023-07-14T19:04:32.249 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.241+0000 7f157e337700 1 -- 172.21.15.78:0/3384009144 wait complete. 2023-07-14T19:04:32.257 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T19:04:32.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.379+0000 7f2bb68a6700 1 -- 172.21.15.78:0/1207262319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb00ffc20 msgr2=0x7f2bb0100030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.379+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/1207262319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb00ffc20 0x7f2bb0100030 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0009990 tx=0x7f2ba0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 -- 172.21.15.78:0/1207262319 shutdown_connections 2023-07-14T19:04:32.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/1207262319 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb01017a0 0x7f2bb0108060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/1207262319 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb00ffc20 0x7f2bb0100030 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/1207262319 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bb0100e10 0x7f2bb0101260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 -- 172.21.15.78:0/1207262319 >> 172.21.15.78:0/1207262319 conn(0x7f2bb00fb260 msgr2=0x7f2bb00fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 -- 172.21.15.78:0/1207262319 shutdown_connections 2023-07-14T19:04:32.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 -- 172.21.15.78:0/1207262319 wait complete. 2023-07-14T19:04:32.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 Processor -- start 2023-07-14T19:04:32.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.380+0000 7f2bb68a6700 1 -- start start 2023-07-14T19:04:32.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb68a6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb00ffc20 0x7f2bb0195890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb68a6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb0100e10 0x7f2bb0195dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb68a6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bb01017a0 0x7f2bb019ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb68a6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb019d410 con 0x7f2bb0100e10 2023-07-14T19:04:32.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb68a6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb019d550 con 0x7f2bb01017a0 2023-07-14T19:04:32.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb68a6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb019d6e0 con 0x7f2bb00ffc20 2023-07-14T19:04:32.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb0100e10 0x7f2bb0195dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb0100e10 0x7f2bb0195dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36366/0 (socket says 172.21.15.78:36366) 2023-07-14T19:04:32.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 -- 172.21.15.78:0/2074178793 learned_addr learned my addr 172.21.15.78:0/2074178793 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:32.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baffff700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb00ffc20 0x7f2bb0195890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb4e43700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bb01017a0 0x7f2bb019ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 -- 172.21.15.78:0/2074178793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb00ffc20 msgr2=0x7f2bb0195890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.392 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb00ffc20 0x7f2bb0195890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 -- 172.21.15.78:0/2074178793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bb01017a0 msgr2=0x7f2bb019ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bb01017a0 0x7f2bb019ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 -- 172.21.15.78:0/2074178793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba00095f0 con 0x7f2bb0100e10 2023-07-14T19:04:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb4e43700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bb01017a0 0x7f2bb019ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:32.393 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2baf7fe700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb0100e10 0x7f2bb0195dd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800e9a0 tx=0x7f2b9800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bad7fa700 1 -- 172.21.15.78:0/2074178793 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98009790 con 0x7f2bb0100e10 2023-07-14T19:04:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bad7fa700 1 -- 172.21.15.78:0/2074178793 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f2b9800cbd0 con 0x7f2bb0100e10 2023-07-14T19:04:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb019d9c0 con 0x7f2bb0100e10 2023-07-14T19:04:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bad7fa700 1 -- 172.21.15.78:0/2074178793 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b98018680 con 0x7f2bb0100e10 2023-07-14T19:04:32.394 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.381+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb019df20 con 0x7f2bb0100e10 2023-07-14T19:04:32.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.382+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb010b2a0 con 0x7f2bb0100e10 2023-07-14T19:04:32.395 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.383+0000 7f2bad7fa700 1 -- 172.21.15.78:0/2074178793 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f2b9800cd40 con 0x7f2bb0100e10 2023-07-14T19:04:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.384+0000 7f2bad7fa700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2b9c0a1260 0x7f2b9c0a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.384+0000 7f2baffff700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2b9c0a1260 0x7f2b9c0a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.384+0000 7f2bad7fa700 1 -- 172.21.15.78:0/2074178793 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f2b98014070 con 0x7f2bb0100e10 2023-07-14T19:04:32.396 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.384+0000 7f2baffff700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2b9c0a1260 0x7f2b9c0a3710 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0006010 tx=0x7f2ba0005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:32.397 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.388+0000 7f2bad7fa700 1 -- 172.21.15.78:0/2074178793 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f2b98051780 con 0x7f2bb0100e10 2023-07-14T19:04:32.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:32 smithi078 ceph-mon[127425]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T19:04:32.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:32 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3353384779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:32 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1503035392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.548 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:32 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3384009144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.585 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.582+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f2bb0106be0 con 0x7f2bb0100e10 2023-07-14T19:04:32.586 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.582+0000 7f2bad7fa700 1 -- 172.21.15.78:0/2074178793 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f2b98099de0 con 0x7f2bb0100e10 2023-07-14T19:04:32.586 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:04:32.587 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2b9c0a1260 msgr2=0x7f2b9c0a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2b9c0a1260 0x7f2b9c0a3710 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2ba0006010 tx=0x7f2ba0005940 comp rx=0 tx=0).stop 2023-07-14T19:04:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb0100e10 msgr2=0x7f2bb0195dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb0100e10 0x7f2bb0195dd0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f2b9800e9a0 tx=0x7f2b9800ed60 comp rx=0 tx=0).stop 2023-07-14T19:04:32.588 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 shutdown_connections 2023-07-14T19:04:32.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f2b9c0a1260 0x7f2b9c0a3710 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2bb01017a0 0x7f2bb019ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2bb00ffc20 0x7f2bb0195890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.589 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 --2- 172.21.15.78:0/2074178793 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2bb0100e10 0x7f2bb0195dd0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 >> 172.21.15.78:0/2074178793 conn(0x7f2bb00fb260 msgr2=0x7f2bb00fd570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:32.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 shutdown_connections 2023-07-14T19:04:32.590 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.584+0000 7f2bb68a6700 1 -- 172.21.15.78:0/2074178793 wait complete. 2023-07-14T19:04:32.600 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T19:04:32.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:32 smithi102 ceph-mon[129280]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T19:04:32.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:32 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3353384779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:32 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1503035392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:32 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3384009144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:32 smithi088 ceph-mon[128973]: osdmap e89: 12 total, 12 up, 12 in 2023-07-14T19:04:32.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:32 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3353384779' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:32 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1503035392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:32 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3384009144' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:32.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.719+0000 7f0c79880700 1 -- 172.21.15.78:0/1474158180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740feb50 msgr2=0x7f0c740fef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.719+0000 7f0c79880700 1 --2- 172.21.15.78:0/1474158180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740feb50 0x7f0c740fef60 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f0c68009990 tx=0x7f0c68009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.719+0000 7f0c79880700 1 -- 172.21.15.78:0/1474158180 shutdown_connections 2023-07-14T19:04:32.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.719+0000 7f0c79880700 1 --2- 172.21.15.78:0/1474158180 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c741006d0 0x7f0c74106f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.719+0000 7f0c79880700 1 --2- 172.21.15.78:0/1474158180 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740feb50 0x7f0c740fef60 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.719+0000 7f0c79880700 1 --2- 172.21.15.78:0/1474158180 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c740ffd40 0x7f0c74100190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.719+0000 7f0c79880700 1 -- 172.21.15.78:0/1474158180 >> 172.21.15.78:0/1474158180 conn(0x7f0c740fa150 msgr2=0x7f0c740fc5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:32.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.720+0000 7f0c79880700 1 -- 172.21.15.78:0/1474158180 shutdown_connections 2023-07-14T19:04:32.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.720+0000 7f0c79880700 1 -- 172.21.15.78:0/1474158180 wait complete. 2023-07-14T19:04:32.727 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.720+0000 7f0c79880700 1 Processor -- start 2023-07-14T19:04:32.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.720+0000 7f0c79880700 1 -- start start 2023-07-14T19:04:32.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.720+0000 7f0c79880700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c740feb50 0x7f0c74069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.720+0000 7f0c79880700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740ffd40 0x7f0c7419a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.728 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.720+0000 7f0c79880700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c741006d0 0x7f0c7419cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c79880700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7419d430 con 0x7f0c740ffd40 2023-07-14T19:04:32.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c79880700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7419d5a0 con 0x7f0c740feb50 2023-07-14T19:04:32.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c79880700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7419d760 con 0x7f0c741006d0 2023-07-14T19:04:32.729 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740ffd40 0x7f0c7419a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740ffd40 0x7f0c7419a600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36380/0 (socket says 172.21.15.78:36380) 2023-07-14T19:04:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 -- 172.21.15.78:0/641083614 learned_addr learned my addr 172.21.15.78:0/641083614 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c737fe700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c741006d0 0x7f0c7419cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c72ffd700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c740feb50 0x7f0c74069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.730 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 -- 172.21.15.78:0/641083614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c741006d0 msgr2=0x7f0c7419cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.731 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c741006d0 0x7f0c7419cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 -- 172.21.15.78:0/641083614 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c740feb50 msgr2=0x7f0c74069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c740feb50 0x7f0c74069560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 -- 172.21.15.78:0/641083614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c680095f0 con 0x7f0c740ffd40 2023-07-14T19:04:32.732 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c72ffd700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c740feb50 0x7f0c74069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c727fc700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740ffd40 0x7f0c7419a600 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400d950 tx=0x7f0c6400dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c7887e700 1 -- 172.21.15.78:0/641083614 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c64009b40 con 0x7f0c740ffd40 2023-07-14T19:04:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c7887e700 1 -- 172.21.15.78:0/641083614 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0c64009ca0 con 0x7f0c740ffd40 2023-07-14T19:04:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7419da40 con 0x7f0c740ffd40 2023-07-14T19:04:32.733 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c7887e700 1 -- 172.21.15.78:0/641083614 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6400f620 con 0x7f0c740ffd40 2023-07-14T19:04:32.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.721+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7419dfd0 con 0x7f0c740ffd40 2023-07-14T19:04:32.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.722+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c74105ad0 con 0x7f0c740ffd40 2023-07-14T19:04:32.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.723+0000 7f0c7887e700 1 -- 172.21.15.78:0/641083614 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0c64010460 con 0x7f0c740ffd40 2023-07-14T19:04:32.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.724+0000 7f0c7887e700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c5c0a11e0 0x7f0c5c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:32.734 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.724+0000 7f0c72ffd700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c5c0a11e0 0x7f0c5c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:32.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.724+0000 7f0c7887e700 1 -- 172.21.15.78:0/641083614 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f0c640cd1d0 con 0x7f0c740ffd40 2023-07-14T19:04:32.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.724+0000 7f0c72ffd700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c5c0a11e0 0x7f0c5c0a3690 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c68000c00 tx=0x7f0c68005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:32.735 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.728+0000 7f0c7887e700 1 -- 172.21.15.78:0/641083614 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0c64096440 con 0x7f0c740ffd40 2023-07-14T19:04:32.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.929+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f0c7402c8e0 con 0x7f0c740ffd40 2023-07-14T19:04:32.933 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.930+0000 7f0c7887e700 1 -- 172.21.15.78:0/641083614 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+588 (secure 0 0 0) 0x7f0c64020260 con 0x7f0c740ffd40 2023-07-14T19:04:32.934 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/378103227 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:32.934 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1800545696 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:32.934 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3249522727 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:32.934 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/378103227 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:32.935 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1797764467 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:32.935 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3249522727 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:32.935 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/21455120 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:32.935 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3838414629 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:32.935 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1459224377 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:32.936 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2696766201 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:32.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c5c0a11e0 msgr2=0x7f0c5c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c5c0a11e0 0x7f0c5c0a3690 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0c68000c00 tx=0x7f0c68005fb0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740ffd40 msgr2=0x7f0c7419a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:32.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740ffd40 0x7f0c7419a600 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f0c6400d950 tx=0x7f0c6400dd10 comp rx=0 tx=0).stop 2023-07-14T19:04:32.937 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 shutdown_connections 2023-07-14T19:04:32.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0c5c0a11e0 0x7f0c5c0a3690 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0c740feb50 0x7f0c74069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c741006d0 0x7f0c7419cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 --2- 172.21.15.78:0/641083614 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0c740ffd40 0x7f0c7419a600 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:32.938 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 >> 172.21.15.78:0/641083614 conn(0x7f0c740fa150 msgr2=0x7f0c740fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:32.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 shutdown_connections 2023-07-14T19:04:32.939 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:32.932+0000 7f0c79880700 1 -- 172.21.15.78:0/641083614 wait complete. 2023-07-14T19:04:32.939 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2023-07-14T19:04:32.948 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2023-07-14T19:04:32.948 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/proc/self/mounts of=/dev/stdout 2023-07-14T19:04:32.966 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:04:33.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.123+0000 7f7a18ac6700 1 -- 172.21.15.78:0/1482014896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a141006c0 msgr2=0x7f7a14106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.123+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/1482014896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a141006c0 0x7f7a14106f80 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00b0a0 tx=0x7f7a0c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 -- 172.21.15.78:0/1482014896 shutdown_connections 2023-07-14T19:04:33.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/1482014896 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a141006c0 0x7f7a14106f80 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/1482014896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a140feb40 0x7f7a140fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/1482014896 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a140ffd30 0x7f7a14100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 -- 172.21.15.78:0/1482014896 >> 172.21.15.78:0/1482014896 conn(0x7f7a140fa160 msgr2=0x7f7a140fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:33.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 -- 172.21.15.78:0/1482014896 shutdown_connections 2023-07-14T19:04:33.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 -- 172.21.15.78:0/1482014896 wait complete. 2023-07-14T19:04:33.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 Processor -- start 2023-07-14T19:04:33.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 -- start start 2023-07-14T19:04:33.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a140feb40 0x7f7a14194810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.124+0000 7f7a18ac6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a140ffd30 0x7f7a14194d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a18ac6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a141006c0 0x7f7a1419bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a18ac6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1419c390 con 0x7f7a140feb40 2023-07-14T19:04:33.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a18ac6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1419c4d0 con 0x7f7a140ffd30 2023-07-14T19:04:33.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a18ac6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1419c660 con 0x7f7a141006c0 2023-07-14T19:04:33.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a140feb40 0x7f7a14194810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a140feb40 0x7f7a14194810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36392/0 (socket says 172.21.15.78:36392) 2023-07-14T19:04:33.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 -- 172.21.15.78:0/2506219666 learned_addr learned my addr 172.21.15.78:0/2506219666 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:33.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a11d9b700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a140ffd30 0x7f7a14194d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a12d9d700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a141006c0 0x7f7a1419bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 -- 172.21.15.78:0/2506219666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a141006c0 msgr2=0x7f7a1419bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a141006c0 0x7f7a1419bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 -- 172.21.15.78:0/2506219666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a140ffd30 msgr2=0x7f7a14194d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a140ffd30 0x7f7a14194d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 -- 172.21.15.78:0/2506219666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a0c009d00 con 0x7f7a140feb40 2023-07-14T19:04:33.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a12d9d700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a141006c0 0x7f7a1419bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:33.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a11d9b700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a140ffd30 0x7f7a14194d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:33.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a1259c700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a140feb40 0x7f7a14194810 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e970 tx=0x7f7a0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:33.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a037fe700 1 -- 172.21.15.78:0/2506219666 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04009790 con 0x7f7a140feb40 2023-07-14T19:04:33.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a037fe700 1 -- 172.21.15.78:0/2506219666 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f7a040044b0 con 0x7f7a140feb40 2023-07-14T19:04:33.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a1419c940 con 0x7f7a140feb40 2023-07-14T19:04:33.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a037fe700 1 -- 172.21.15.78:0/2506219666 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04003dd0 con 0x7f7a140feb40 2023-07-14T19:04:33.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.125+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a1419ce50 con 0x7f7a140feb40 2023-07-14T19:04:33.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.126+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a1410a1d0 con 0x7f7a140feb40 2023-07-14T19:04:33.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.127+0000 7f7a037fe700 1 -- 172.21.15.78:0/2506219666 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f7a04004000 con 0x7f7a140feb40 2023-07-14T19:04:33.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.128+0000 7f7a037fe700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f79fc0a1220 0x7f79fc0a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.128+0000 7f7a11d9b700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f79fc0a1220 0x7f79fc0a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.128+0000 7f7a037fe700 1 -- 172.21.15.78:0/2506219666 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f7a04014070 con 0x7f7a140feb40 2023-07-14T19:04:33.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.128+0000 7f7a11d9b700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f79fc0a1220 0x7f79fc0a36d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009cb0 tx=0x7f7a08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:33.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.131+0000 7f7a037fe700 1 -- 172.21.15.78:0/2506219666 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f7a040964a0 con 0x7f7a140feb40 2023-07-14T19:04:33.322 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.319+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a14105980 con 0x7f7a140feb40 2023-07-14T19:04:33.324 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.321+0000 7f7a037fe700 1 -- 172.21.15.78:0/2506219666 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24640 (secure 0 0 0) 0x7f7a04099ac0 con 0x7f7a140feb40 2023-07-14T19:04:33.325 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:33.325 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:04:31.284921+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:04:33.326 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f79fc0a1220 msgr2=0x7f79fc0a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f79fc0a1220 0x7f79fc0a36d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009cb0 tx=0x7f7a08009260 comp rx=0 tx=0).stop 2023-07-14T19:04:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a140feb40 msgr2=0x7f7a14194810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a140feb40 0x7f7a14194810 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400e970 tx=0x7f7a0400ed30 comp rx=0 tx=0).stop 2023-07-14T19:04:33.327 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 shutdown_connections 2023-07-14T19:04:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f79fc0a1220 0x7f79fc0a36d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a141006c0 0x7f7a1419bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f7a140feb40 0x7f7a14194810 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 --2- 172.21.15.78:0/2506219666 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f7a140ffd30 0x7f7a14194d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.328 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 >> 172.21.15.78:0/2506219666 conn(0x7f7a140fa160 msgr2=0x7f7a140fc330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 shutdown_connections 2023-07-14T19:04:33.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.323+0000 7f7a18ac6700 1 -- 172.21.15.78:0/2506219666 wait complete. 2023-07-14T19:04:33.338 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-07-14T18:58:40.361042+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-07-14T19:02:28.269050+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '80', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-07-14T19:02:29.547771+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2023-07-14T19:02:32.797605+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2023-07-14T19:04:33.339 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-07-14T19:04:33.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 -- 172.21.15.78:0/1801632892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 msgr2=0x7f1314106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.464 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1801632892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 0x7f1314106f50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f130c00b0a0 tx=0x7f130c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 -- 172.21.15.78:0/1801632892 shutdown_connections 2023-07-14T19:04:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1801632892 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 0x7f1314106f50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1801632892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13140feb10 0x7f13140fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1801632892 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13140ffd00 0x7f1314100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.465 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 -- 172.21.15.78:0/1801632892 >> 172.21.15.78:0/1801632892 conn(0x7f13140fa130 msgr2=0x7f13140fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:33.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 -- 172.21.15.78:0/1801632892 shutdown_connections 2023-07-14T19:04:33.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 -- 172.21.15.78:0/1801632892 wait complete. 2023-07-14T19:04:33.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.460+0000 7f131a6bd700 1 Processor -- start 2023-07-14T19:04:33.466 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f131a6bd700 1 -- start start 2023-07-14T19:04:33.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f131a6bd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13140feb10 0x7f1314102aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f131a6bd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13140ffd00 0x7f13141010e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f131a6bd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 0x7f1314101620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f131a6bd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314101dc0 con 0x7f1314100690 2023-07-14T19:04:33.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f131a6bd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1314101f30 con 0x7f13140feb10 2023-07-14T19:04:33.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f131a6bd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f131419fa20 con 0x7f13140ffd00 2023-07-14T19:04:33.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1318c5a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 0x7f1314101620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1318c5a700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 0x7f1314101620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36410/0 (socket says 172.21.15.78:36410) 2023-07-14T19:04:33.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1318c5a700 1 -- 172.21.15.78:0/1224094746 learned_addr learned my addr 172.21.15.78:0/1224094746 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:33.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f13137fe700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13140ffd00 0x7f13141010e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1313fff700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13140feb10 0x7f1314102aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.470 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1318c5a700 1 -- 172.21.15.78:0/1224094746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13140ffd00 msgr2=0x7f13141010e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1318c5a700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13140ffd00 0x7f13141010e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1318c5a700 1 -- 172.21.15.78:0/1224094746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13140feb10 msgr2=0x7f1314102aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1318c5a700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13140feb10 0x7f1314102aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.461+0000 7f1318c5a700 1 -- 172.21.15.78:0/1224094746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c009d00 con 0x7f1314100690 2023-07-14T19:04:33.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.462+0000 7f1318c5a700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 0x7f1314101620 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f130c00b070 tx=0x7f130c004c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:33.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.462+0000 7f13117fa700 1 -- 172.21.15.78:0/1224094746 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c00e040 con 0x7f1314100690 2023-07-14T19:04:33.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.462+0000 7f13117fa700 1 -- 172.21.15.78:0/1224094746 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f130c003e90 con 0x7f1314100690 2023-07-14T19:04:33.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.462+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f131419fca0 con 0x7f1314100690 2023-07-14T19:04:33.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.462+0000 7f13117fa700 1 -- 172.21.15.78:0/1224094746 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f130c004020 con 0x7f1314100690 2023-07-14T19:04:33.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.462+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13141a00d0 con 0x7f1314100690 2023-07-14T19:04:33.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.463+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f131410a1f0 con 0x7f1314100690 2023-07-14T19:04:33.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.464+0000 7f13117fa700 1 -- 172.21.15.78:0/1224094746 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f130c003a00 con 0x7f1314100690 2023-07-14T19:04:33.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.464+0000 7f13117fa700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12fc0a11e0 0x7f12fc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.464+0000 7f13117fa700 1 -- 172.21.15.78:0/1224094746 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f130c0cef90 con 0x7f1314100690 2023-07-14T19:04:33.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.464+0000 7f1313fff700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12fc0a11e0 0x7f12fc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.465+0000 7f1313fff700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12fc0a11e0 0x7f12fc0a3690 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13140ffb60 tx=0x7f130400b5b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:33.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.467+0000 7f13117fa700 1 -- 172.21.15.78:0/1224094746 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f130c098290 con 0x7f1314100690 2023-07-14T19:04:33.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:33 smithi102 ceph-mon[129280]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-07-14T19:04:33.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:33 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2074178793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:33.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:33 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/641083614' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:04:33.657 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.654+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13141a0680 con 0x7f1314100690 2023-07-14T19:04:33.658 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.654+0000 7f13117fa700 1 -- 172.21.15.78:0/1224094746 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f130c0243b0 con 0x7f1314100690 2023-07-14T19:04:33.658 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2023-07-14T19:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.656+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12fc0a11e0 msgr2=0x7f12fc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.656+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12fc0a11e0 0x7f12fc0a3690 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f13140ffb60 tx=0x7f130400b5b0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.660 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 msgr2=0x7f1314101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 0x7f1314101620 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f130c00b070 tx=0x7f130c004c80 comp rx=0 tx=0).stop 2023-07-14T19:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 shutdown_connections 2023-07-14T19:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f12fc0a11e0 0x7f12fc0a3690 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.661 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f13140feb10 0x7f1314102aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f1314100690 0x7f1314101620 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 --2- 172.21.15.78:0/1224094746 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f13140ffd00 0x7f13141010e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 >> 172.21.15.78:0/1224094746 conn(0x7f13140fa130 msgr2=0x7f1314067860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 shutdown_connections 2023-07-14T19:04:33.662 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.657+0000 7f131a6bd700 1 -- 172.21.15.78:0/1224094746 wait complete. 2023-07-14T19:04:33.671 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:33 smithi078 ceph-mon[127425]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-07-14T19:04:33.671 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:33 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2074178793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:33.672 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:33 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/641083614' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:04:33.673 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-07-14T19:04:33.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:33 smithi088 ceph-mon[128973]: pgmap v320: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 1.4 KiB/s wr, 7 op/s 2023-07-14T19:04:33.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:33 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2074178793' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:33.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:33 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/641083614' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:04:33.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 -- 172.21.15.78:0/812196891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24740ffc10 msgr2=0x7f2474100020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 --2- 172.21.15.78:0/812196891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24740ffc10 0x7f2474100020 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2468009990 tx=0x7f2468009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.797 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 -- 172.21.15.78:0/812196891 shutdown_connections 2023-07-14T19:04:33.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 --2- 172.21.15.78:0/812196891 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f2474101790 0x7f2474108050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 --2- 172.21.15.78:0/812196891 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f24740ffc10 0x7f2474100020 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 --2- 172.21.15.78:0/812196891 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2474100e00 0x7f2474101250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 -- 172.21.15.78:0/812196891 >> 172.21.15.78:0/812196891 conn(0x7f24740fb250 msgr2=0x7f24740fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:33.798 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 -- 172.21.15.78:0/812196891 shutdown_connections 2023-07-14T19:04:33.799 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.793+0000 7f247ad99700 1 -- 172.21.15.78:0/812196891 wait complete. 2023-07-14T19:04:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.794+0000 7f247ad99700 1 Processor -- start 2023-07-14T19:04:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.794+0000 7f247ad99700 1 -- start start 2023-07-14T19:04:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.794+0000 7f247ad99700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24740ffc10 0x7f24741958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.801 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.794+0000 7f247ad99700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2474100e00 0x7f2474195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.794+0000 7f247ad99700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2474101790 0x7f247419ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.794+0000 7f247ad99700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247419d450 con 0x7f2474101790 2023-07-14T19:04:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.794+0000 7f247ad99700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247419d590 con 0x7f24740ffc10 2023-07-14T19:04:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.794+0000 7f247ad99700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f247419d720 con 0x7f2474100e00 2023-07-14T19:04:33.802 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2474101790 0x7f247419ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2474101790 0x7f247419ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36424/0 (socket says 172.21.15.78:36424) 2023-07-14T19:04:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 -- 172.21.15.78:0/473782209 learned_addr learned my addr 172.21.15.78:0/473782209 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2478b35700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24740ffc10 0x7f24741958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2473fff700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2474100e00 0x7f2474195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.803 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 -- 172.21.15.78:0/473782209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2474100e00 msgr2=0x7f2474195e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2474100e00 0x7f2474195e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 -- 172.21.15.78:0/473782209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24740ffc10 msgr2=0x7f24741958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24740ffc10 0x7f24741958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.804 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 -- 172.21.15.78:0/473782209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24680095f0 con 0x7f2474101790 2023-07-14T19:04:33.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2473fff700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2474100e00 0x7f2474195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:33.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2479336700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2474101790 0x7f247419ce80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f246c00ada0 tx=0x7f246c00c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:33.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2471ffb700 1 -- 172.21.15.78:0/473782209 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c00f070 con 0x7f2474101790 2023-07-14T19:04:33.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2471ffb700 1 -- 172.21.15.78:0/473782209 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f246c007b30 con 0x7f2474101790 2023-07-14T19:04:33.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f2471ffb700 1 -- 172.21.15.78:0/473782209 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f246c005280 con 0x7f2474101790 2023-07-14T19:04:33.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f247419da00 con 0x7f2474101790 2023-07-14T19:04:33.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.795+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f247419df40 con 0x7f2474101790 2023-07-14T19:04:33.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.796+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f247410b310 con 0x7f2474101790 2023-07-14T19:04:33.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.797+0000 7f2471ffb700 1 -- 172.21.15.78:0/473782209 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f246c00c980 con 0x7f2474101790 2023-07-14T19:04:33.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.798+0000 7f2471ffb700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f24640a1260 0x7f24640a3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:33.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.798+0000 7f2478b35700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f24640a1260 0x7f24640a3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:33.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.798+0000 7f2471ffb700 1 -- 172.21.15.78:0/473782209 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f246c0cc4f0 con 0x7f2474101790 2023-07-14T19:04:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.798+0000 7f2478b35700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f24640a1260 0x7f24640a3710 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f246800b5c0 tx=0x7f2468005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:33.808 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.802+0000 7f2471ffb700 1 -- 172.21.15.78:0/473782209 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f246c0957b0 con 0x7f2474101790 2023-07-14T19:04:33.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.989+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f2474106be0 con 0x7f2474101790 2023-07-14T19:04:33.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.989+0000 7f2471ffb700 1 -- 172.21.15.78:0/473782209 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f246c00cc30 con 0x7f2474101790 2023-07-14T19:04:33.993 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:04:33.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f24640a1260 msgr2=0x7f24640a3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f24640a1260 0x7f24640a3710 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f246800b5c0 tx=0x7f2468005960 comp rx=0 tx=0).stop 2023-07-14T19:04:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2474101790 msgr2=0x7f247419ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2474101790 0x7f247419ce80 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f246c00ada0 tx=0x7f246c00c330 comp rx=0 tx=0).stop 2023-07-14T19:04:33.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 shutdown_connections 2023-07-14T19:04:33.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f24640a1260 0x7f24640a3710 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f24740ffc10 0x7f24741958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f2474101790 0x7f247419ce80 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 --2- 172.21.15.78:0/473782209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2474100e00 0x7f2474195e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:33.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 >> 172.21.15.78:0/473782209 conn(0x7f24740fb250 msgr2=0x7f24740fd590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 shutdown_connections 2023-07-14T19:04:33.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:33.991+0000 7f247ad99700 1 -- 172.21.15.78:0/473782209 wait complete. 2023-07-14T19:04:34.007 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-07-14T19:04:34.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 -- 172.21.15.78:0/2652305366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80fee00 msgr2=0x7f9ef80ff270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/2652305366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80fee00 0x7f9ef80ff270 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8009990 tx=0x7f9ee8009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 -- 172.21.15.78:0/2652305366 shutdown_connections 2023-07-14T19:04:34.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/2652305366 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ef81001b0 0x7f9ef81005c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/2652305366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ef80ff7b0 0x7f9ef8107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/2652305366 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80fee00 0x7f9ef80ff270 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 -- 172.21.15.78:0/2652305366 >> 172.21.15.78:0/2652305366 conn(0x7f9ef80f9980 msgr2=0x7f9ef80fbdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 -- 172.21.15.78:0/2652305366 shutdown_connections 2023-07-14T19:04:34.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.127+0000 7f9efd2b2700 1 -- 172.21.15.78:0/2652305366 wait complete. 2023-07-14T19:04:34.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9efd2b2700 1 Processor -- start 2023-07-14T19:04:34.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9efd2b2700 1 -- start start 2023-07-14T19:04:34.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9efd2b2700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ef80fee00 0x7f9ef8195920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9efd2b2700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80ff7b0 0x7f9ef8195e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9efd2b2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ef81001b0 0x7f9ef819ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.135 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9efd2b2700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef819d4a0 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9efd2b2700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef819d5e0 con 0x7f9ef80fee00 2023-07-14T19:04:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9efd2b2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef819d770 con 0x7f9ef81001b0 2023-07-14T19:04:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80ff7b0 0x7f9ef8195e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80ff7b0 0x7f9ef8195e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36436/0 (socket says 172.21.15.78:36436) 2023-07-14T19:04:34.136 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 -- 172.21.15.78:0/123632830 learned_addr learned my addr 172.21.15.78:0/123632830 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:34.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef6ffd700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ef80fee00 0x7f9ef8195920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef77fe700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ef81001b0 0x7f9ef819ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 -- 172.21.15.78:0/123632830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ef81001b0 msgr2=0x7f9ef819ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ef81001b0 0x7f9ef819ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.137 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 -- 172.21.15.78:0/123632830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ef80fee00 msgr2=0x7f9ef8195920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ef80fee00 0x7f9ef8195920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 -- 172.21.15.78:0/123632830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee80095f0 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.128+0000 7f9ef67fc700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80ff7b0 0x7f9ef8195e60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80098e0 tx=0x7f9ee8005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.129+0000 7f9eeffff700 1 -- 172.21.15.78:0/123632830 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee801d070 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.138 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.129+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef819d9f0 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.129+0000 7f9eeffff700 1 -- 172.21.15.78:0/123632830 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f9ee800f460 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.129+0000 7f9eeffff700 1 -- 172.21.15.78:0/123632830 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ee8021670 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.129+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef819de70 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.130+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef8106da0 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.131+0000 7f9eeffff700 1 -- 172.21.15.78:0/123632830 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f9ee800f5d0 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.139 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.131+0000 7f9eeffff700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9ee40a11e0 0x7f9ee40a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.131+0000 7f9ef6ffd700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9ee40a11e0 0x7f9ee40a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.140 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.131+0000 7f9eeffff700 1 -- 172.21.15.78:0/123632830 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f9ee80ce320 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.132+0000 7f9ef6ffd700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9ee40a11e0 0x7f9ee40a3690 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0005fd0 tx=0x7f9ee0005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:34.141 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.135+0000 7f9eeffff700 1 -- 172.21.15.78:0/123632830 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f9ee8097620 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.325+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9ef802c8e0 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.329 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.326+0000 7f9eeffff700 1 -- 172.21.15.78:0/123632830 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9ee809ac40 con 0x7f9ef80ff7b0 2023-07-14T19:04:34.329 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 64 2023-07-14T19:04:34.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9ee40a11e0 msgr2=0x7f9ee40a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.331 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9ee40a11e0 0x7f9ee40a3690 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f9ee0005fd0 tx=0x7f9ee0005dd0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80ff7b0 msgr2=0x7f9ef8195e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80ff7b0 0x7f9ef8195e60 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f9ee80098e0 tx=0x7f9ee8005ef0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 shutdown_connections 2023-07-14T19:04:34.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f9ee40a11e0 0x7f9ee40a3690 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.332 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f9ef80fee00 0x7f9ef8195920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ef81001b0 0x7f9ef819ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 --2- 172.21.15.78:0/123632830 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f9ef80ff7b0 0x7f9ef8195e60 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 >> 172.21.15.78:0/123632830 conn(0x7f9ef80f9980 msgr2=0x7f9ef80fbc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:34.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 shutdown_connections 2023-07-14T19:04:34.333 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.328+0000 7f9efd2b2700 1 -- 172.21.15.78:0/123632830 wait complete. 2023-07-14T19:04:34.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2023-07-14T19:04:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 -- 172.21.15.78:0/1533319781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e500ffd20 msgr2=0x7f0e50100170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1533319781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e500ffd20 0x7f0e50100170 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c009990 tx=0x7f0e4c009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.467 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 -- 172.21.15.78:0/1533319781 shutdown_connections 2023-07-14T19:04:34.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1533319781 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e500feb30 0x7f0e500fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1533319781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e501006b0 0x7f0e50106f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1533319781 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e500ffd20 0x7f0e50100170 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 -- 172.21.15.78:0/1533319781 >> 172.21.15.78:0/1533319781 conn(0x7f0e500fa150 msgr2=0x7f0e500fc580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:34.468 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 -- 172.21.15.78:0/1533319781 shutdown_connections 2023-07-14T19:04:34.469 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.463+0000 7f0e586e6700 1 -- 172.21.15.78:0/1533319781 wait complete. 2023-07-14T19:04:34.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e586e6700 1 Processor -- start 2023-07-14T19:04:34.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e586e6700 1 -- start start 2023-07-14T19:04:34.471 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e586e6700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e500feb30 0x7f0e50069560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e586e6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e500ffd20 0x7f0e5019a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e586e6700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e501006b0 0x7f0e5019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e586e6700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5019d3e0 con 0x7f0e501006b0 2023-07-14T19:04:34.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e586e6700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5019d550 con 0x7f0e500feb30 2023-07-14T19:04:34.472 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e586e6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5019d710 con 0x7f0e500ffd20 2023-07-14T19:04:34.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e56c83700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e501006b0 0x7f0e5019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e56c83700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e501006b0 0x7f0e5019cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36452/0 (socket says 172.21.15.78:36452) 2023-07-14T19:04:34.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e56c83700 1 -- 172.21.15.78:0/1324763032 learned_addr learned my addr 172.21.15.78:0/1324763032 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:34.473 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e55c81700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e500ffd20 0x7f0e5019a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.464+0000 7f0e56482700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e500feb30 0x7f0e50069560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e56c83700 1 -- 172.21.15.78:0/1324763032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e500ffd20 msgr2=0x7f0e5019a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.474 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e56c83700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e500ffd20 0x7f0e5019a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e56c83700 1 -- 172.21.15.78:0/1324763032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e500feb30 msgr2=0x7f0e50069560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e56c83700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e500feb30 0x7f0e50069560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e56c83700 1 -- 172.21.15.78:0/1324763032 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e4c0095f0 con 0x7f0e501006b0 2023-07-14T19:04:34.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e56482700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e500feb30 0x7f0e50069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:34.475 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e56c83700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e501006b0 0x7f0e5019cc10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0e48007ee0 tx=0x7f0e4800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:34.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e437fe700 1 -- 172.21.15.78:0/1324763032 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48017070 con 0x7f0e501006b0 2023-07-14T19:04:34.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e437fe700 1 -- 172.21.15.78:0/1324763032 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f0e4800f040 con 0x7f0e501006b0 2023-07-14T19:04:34.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5019d9f0 con 0x7f0e501006b0 2023-07-14T19:04:34.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e437fe700 1 -- 172.21.15.78:0/1324763032 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e48005710 con 0x7f0e501006b0 2023-07-14T19:04:34.476 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.465+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5019ded0 con 0x7f0e501006b0 2023-07-14T19:04:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.466+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e50105ab0 con 0x7f0e501006b0 2023-07-14T19:04:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.467+0000 7f0e437fe700 1 -- 172.21.15.78:0/1324763032 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f0e48005870 con 0x7f0e501006b0 2023-07-14T19:04:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.467+0000 7f0e437fe700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e3c0a11e0 0x7f0e3c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.467+0000 7f0e56482700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e3c0a11e0 0x7f0e3c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.477 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.467+0000 7f0e437fe700 1 -- 172.21.15.78:0/1324763032 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f0e480cc630 con 0x7f0e501006b0 2023-07-14T19:04:34.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.468+0000 7f0e56482700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e3c0a11e0 0x7f0e3c0a3690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0e44005950 tx=0x7f0e440058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:34.478 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.472+0000 7f0e437fe700 1 -- 172.21.15.78:0/1324763032 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f0e480958a0 con 0x7f0e501006b0 2023-07-14T19:04:34.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:34 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2506219666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:34.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:34 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1224094746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:34.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:34 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/473782209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:34.668 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.665+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0e5002cb30 con 0x7f0e501006b0 2023-07-14T19:04:34.669 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.665+0000 7f0e437fe700 1 -- 172.21.15.78:0/1324763032 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f0e48095020 con 0x7f0e501006b0 2023-07-14T19:04:34.669 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 32 2023-07-14T19:04:34.670 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e3c0a11e0 msgr2=0x7f0e3c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e3c0a11e0 0x7f0e3c0a3690 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0e44005950 tx=0x7f0e440058e0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e501006b0 msgr2=0x7f0e5019cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e501006b0 0x7f0e5019cc10 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0e48007ee0 tx=0x7f0e4800d3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 shutdown_connections 2023-07-14T19:04:34.671 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f0e3c0a11e0 0x7f0e3c0a3690 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f0e500feb30 0x7f0e50069560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f0e501006b0 0x7f0e5019cc10 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 --2- 172.21.15.78:0/1324763032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e500ffd20 0x7f0e5019a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.672 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 >> 172.21.15.78:0/1324763032 conn(0x7f0e500fa150 msgr2=0x7f0e500fc420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 shutdown_connections 2023-07-14T19:04:34.673 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.667+0000 7f0e586e6700 1 -- 172.21.15.78:0/1324763032 wait complete. 2023-07-14T19:04:34.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:34 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2506219666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:34.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:34 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1224094746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:34.682 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:34 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/473782209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:34.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-07-14T19:04:34.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:34 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2506219666' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:34.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:34 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1224094746' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-07-14T19:04:34.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:34 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/473782209' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2023-07-14T19:04:34.805 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 -- 172.21.15.78:0/1673922787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f941006c0 msgr2=0x7f4f94106f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/1673922787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f941006c0 0x7f4f94106f80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c00b0a0 tx=0x7f4f8c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 -- 172.21.15.78:0/1673922787 shutdown_connections 2023-07-14T19:04:34.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/1673922787 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f940feb40 0x7f4f940fef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.806 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/1673922787 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f941006c0 0x7f4f94106f80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/1673922787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f940ffd30 0x7f4f94100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 -- 172.21.15.78:0/1673922787 >> 172.21.15.78:0/1673922787 conn(0x7f4f940fa160 msgr2=0x7f4f940fc590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:34.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 -- 172.21.15.78:0/1673922787 shutdown_connections 2023-07-14T19:04:34.807 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.802+0000 7f4f98f0e700 1 -- 172.21.15.78:0/1673922787 wait complete. 2023-07-14T19:04:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f98f0e700 1 Processor -- start 2023-07-14T19:04:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f98f0e700 1 -- start start 2023-07-14T19:04:34.809 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f98f0e700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940feb40 0x7f4f94102ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f98f0e700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f940ffd30 0x7f4f94101110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f98f0e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f941006c0 0x7f4f94101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f98f0e700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f94101df0 con 0x7f4f940feb40 2023-07-14T19:04:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f98f0e700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f94101f60 con 0x7f4f940ffd30 2023-07-14T19:04:34.810 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f98f0e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f9419f9b0 con 0x7f4f941006c0 2023-07-14T19:04:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940feb40 0x7f4f94102ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940feb40 0x7f4f94102ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36462/0 (socket says 172.21.15.78:36462) 2023-07-14T19:04:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 -- 172.21.15.78:0/2624801546 learned_addr learned my addr 172.21.15.78:0/2624801546 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f92d9d700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f941006c0 0x7f4f94101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.811 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f91d9b700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f940ffd30 0x7f4f94101110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 -- 172.21.15.78:0/2624801546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f941006c0 msgr2=0x7f4f94101650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f941006c0 0x7f4f94101650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 -- 172.21.15.78:0/2624801546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f940ffd30 msgr2=0x7f4f94101110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f940ffd30 0x7f4f94101110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:34.812 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 -- 172.21.15.78:0/2624801546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f8c009d00 con 0x7f4f940feb40 2023-07-14T19:04:34.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.803+0000 7f4f9259c700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940feb40 0x7f4f94102ad0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400d980 tx=0x7f4f8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:34.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.804+0000 7f4f837fe700 1 -- 172.21.15.78:0/2624801546 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f8400b950 con 0x7f4f940feb40 2023-07-14T19:04:34.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.804+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f9419fc90 con 0x7f4f940feb40 2023-07-14T19:04:34.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.804+0000 7f4f837fe700 1 -- 172.21.15.78:0/2624801546 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f4f84007620 con 0x7f4f940feb40 2023-07-14T19:04:34.813 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.804+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f941a01a0 con 0x7f4f940feb40 2023-07-14T19:04:34.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.804+0000 7f4f837fe700 1 -- 172.21.15.78:0/2624801546 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f84004830 con 0x7f4f940feb40 2023-07-14T19:04:34.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.805+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f9410a1d0 con 0x7f4f940feb40 2023-07-14T19:04:34.814 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.806+0000 7f4f837fe700 1 -- 172.21.15.78:0/2624801546 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f4f8400bab0 con 0x7f4f940feb40 2023-07-14T19:04:34.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.806+0000 7f4f837fe700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4f7c0a11e0 0x7f4f7c0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:34.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.806+0000 7f4f837fe700 1 -- 172.21.15.78:0/2624801546 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f4f840cd4d0 con 0x7f4f940feb40 2023-07-14T19:04:34.815 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.806+0000 7f4f91d9b700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4f7c0a11e0 0x7f4f7c0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:34.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.807+0000 7f4f91d9b700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4f7c0a11e0 0x7f4f7c0a3690 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4f88005950 tx=0x7f4f880058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:34.816 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:34.810+0000 7f4f837fe700 1 -- 172.21.15.78:0/2624801546 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f4f84096740 con 0x7f4f940feb40 2023-07-14T19:04:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.015+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f4f9410a3c0 con 0x7f4f940feb40 2023-07-14T19:04:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.015+0000 7f4f837fe700 1 -- 172.21.15.78:0/2624801546 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+588 (secure 0 0 0) 0x7f4f8400bd60 con 0x7f4f940feb40 2023-07-14T19:04:35.020 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/378103227 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:35.020 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1800545696 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:35.021 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6800/3249522727 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:35.021 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/378103227 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:35.021 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1797764467 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:35.021 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:6801/3249522727 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:35.022 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/21455120 2023-07-15T18:56:30.036843+0000 2023-07-14T19:04:35.022 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/3838414629 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:35.022 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/1459224377 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:35.022 INFO:teuthology.orchestra.run.smithi078.stdout:172.21.15.78:0/2696766201 2023-07-15T18:56:08.189962+0000 2023-07-14T19:04:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.017+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4f7c0a11e0 msgr2=0x7f4f7c0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.017+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4f7c0a11e0 0x7f4f7c0a3690 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4f88005950 tx=0x7f4f880058e0 comp rx=0 tx=0).stop 2023-07-14T19:04:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.017+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940feb40 msgr2=0x7f4f94102ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.017+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940feb40 0x7f4f94102ad0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400d980 tx=0x7f4f8400dc90 comp rx=0 tx=0).stop 2023-07-14T19:04:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.018+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 shutdown_connections 2023-07-14T19:04:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.018+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f4f7c0a11e0 0x7f4f7c0a3690 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.018+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f941006c0 0x7f4f94101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.018+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f4f940feb40 0x7f4f94102ad0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.018+0000 7f4f98f0e700 1 --2- 172.21.15.78:0/2624801546 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f4f940ffd30 0x7f4f94101110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.018+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 >> 172.21.15.78:0/2624801546 conn(0x7f4f940fa160 msgr2=0x7f4f94068830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.018+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 shutdown_connections 2023-07-14T19:04:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:35.018+0000 7f4f98f0e700 1 -- 172.21.15.78:0/2624801546 wait complete. 2023-07-14T19:04:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:listed 10 entries 2023-07-14T19:04:35.033 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi078.front.sepia.ceph.com... 2023-07-14T19:04:35.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:35.034 DEBUG:teuthology.orchestra.run.smithi078:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2023-07-14T19:04:35.068 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi078.stderr:ceph-fuse[178281]: fuse finished with error 0 and tester_r 0 2023-07-14T19:04:35.070 INFO:teuthology.orchestra.run:waiting for 300 2023-07-14T19:04:35.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:35 smithi102 ceph-mon[129280]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-07-14T19:04:35.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:35 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/123632830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:35.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:35 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1324763032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:35.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:35 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2624801546' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:04:35.679 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:35 smithi078 ceph-mon[127425]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-07-14T19:04:35.680 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:35 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/123632830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:35.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:35 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1324763032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:35.681 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:35 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2624801546' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:04:35.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:35 smithi088 ceph-mon[128973]: pgmap v321: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2023-07-14T19:04:35.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:35 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/123632830' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2023-07-14T19:04:35.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:35 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1324763032' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2023-07-14T19:04:35.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:35 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2624801546' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2023-07-14T19:04:36.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:36 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:04:36.962 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:36 smithi088 ceph-mon[128973]: pgmap v322: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 6 op/s 2023-07-14T19:04:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:36 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:04:37.047 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:36 smithi078 ceph-mon[127425]: pgmap v322: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 6 op/s 2023-07-14T19:04:37.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:36 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-07-14T19:04:37.131 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:36 smithi102 ceph-mon[129280]: pgmap v322: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 767 B/s wr, 6 op/s 2023-07-14T19:04:39.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: pgmap v323: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:04:39.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:04:39.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:04:39.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:39 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: pgmap v323: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:04:39.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:04:39.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:04:39.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:39 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: pgmap v323: 129 pgs: 129 active+clean; 643 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 5 op/s 2023-07-14T19:04:39.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:39.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-07-14T19:04:39.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-07-14T19:04:39.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:39 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' 2023-07-14T19:04:41.172 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi078.front.sepia.ceph.com 2023-07-14T19:04:41.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:41.174 DEBUG:teuthology.orchestra.run.smithi078:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-07-14T19:04:41.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:41.198 DEBUG:teuthology.orchestra.run.smithi078:> ip netns list 2023-07-14T19:04:41.259 INFO:teuthology.orchestra.run.smithi078.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-07-14T19:04:41.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:41.261 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-07-14T19:04:41.352 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-07-14T19:04:41.352 DEBUG:teuthology.orchestra.run.smithi078:> sudo ip link delete ceph-brx 2023-07-14T19:04:41.469 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-07-14T19:04:41.479 INFO:tasks.cephadm:Teardown begin 2023-07-14T19:04:41.480 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-07-14T19:04:41.480 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:04:41.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 -- 172.21.15.78:0/1909138776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 msgr2=0x7f691c1013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:41.609 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 --2- 172.21.15.78:0/1909138776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 0x7f691c1013a0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f6910009990 tx=0x7f6910009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 -- 172.21.15.78:0/1909138776 shutdown_connections 2023-07-14T19:04:41.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 --2- 172.21.15.78:0/1909138776 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f691c0fdee0 0x7f691c0fe2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 --2- 172.21.15.78:0/1909138776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f691c1018e0 0x7f691c1081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 --2- 172.21.15.78:0/1909138776 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 0x7f691c1013a0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.610 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 -- 172.21.15.78:0/1909138776 >> 172.21.15.78:0/1909138776 conn(0x7f691c0f98a0 msgr2=0x7f691c0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 -- 172.21.15.78:0/1909138776 shutdown_connections 2023-07-14T19:04:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.605+0000 7f6921f3d700 1 -- 172.21.15.78:0/1909138776 wait complete. 2023-07-14T19:04:41.611 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f6921f3d700 1 Processor -- start 2023-07-14T19:04:41.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f6921f3d700 1 -- start start 2023-07-14T19:04:41.613 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f6921f3d700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f691c0fdee0 0x7f691c1957b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f6921f3d700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 0x7f691c195cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f6921f3d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f691c1018e0 0x7f691c19cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f6921f3d700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c19d330 con 0x7f691c100f50 2023-07-14T19:04:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f6921f3d700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c19d470 con 0x7f691c0fdee0 2023-07-14T19:04:41.614 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f6921f3d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f691c19d600 con 0x7f691c1018e0 2023-07-14T19:04:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f691affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 0x7f691c195cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f691affd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 0x7f691c195cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36478/0 (socket says 172.21.15.78:36478) 2023-07-14T19:04:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.606+0000 7f691affd700 1 -- 172.21.15.78:0/330848790 learned_addr learned my addr 172.21.15.78:0/330848790 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691bfff700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f691c1018e0 0x7f691c19cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:41.615 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691b7fe700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f691c0fdee0 0x7f691c1957b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691affd700 1 -- 172.21.15.78:0/330848790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f691c1018e0 msgr2=0x7f691c19cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691affd700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f691c1018e0 0x7f691c19cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691affd700 1 -- 172.21.15.78:0/330848790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f691c0fdee0 msgr2=0x7f691c1957b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691affd700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f691c0fdee0 0x7f691c1957b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.616 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691affd700 1 -- 172.21.15.78:0/330848790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69100095f0 con 0x7f691c100f50 2023-07-14T19:04:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691b7fe700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f691c0fdee0 0x7f691c1957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f691affd700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 0x7f691c195cf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6910006010 tx=0x7f6910003b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f6918ff9700 1 -- 172.21.15.78:0/330848790 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691001d070 con 0x7f691c100f50 2023-07-14T19:04:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f6918ff9700 1 -- 172.21.15.78:0/330848790 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7f6910004640 con 0x7f691c100f50 2023-07-14T19:04:41.617 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f691c19d880 con 0x7f691c100f50 2023-07-14T19:04:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f6918ff9700 1 -- 172.21.15.78:0/330848790 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f691000f460 con 0x7f691c100f50 2023-07-14T19:04:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.607+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f691c19dd20 con 0x7f691c100f50 2023-07-14T19:04:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.608+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f691c107280 con 0x7f691c100f50 2023-07-14T19:04:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.609+0000 7f6918ff9700 1 -- 172.21.15.78:0/330848790 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7f6910022470 con 0x7f691c100f50 2023-07-14T19:04:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.609+0000 7f6918ff9700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f69040a1220 0x7f69040a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:41.618 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.610+0000 7f691b7fe700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f69040a1220 0x7f69040a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.610+0000 7f6918ff9700 1 -- 172.21.15.78:0/330848790 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7f69100ce380 con 0x7f691c100f50 2023-07-14T19:04:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.610+0000 7f691b7fe700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f69040a1220 0x7f69040a36d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f690c005fd0 tx=0x7f690c005e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:41.619 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.614+0000 7f6918ff9700 1 -- 172.21.15.78:0/330848790 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7f691005c780 con 0x7f691c100f50 2023-07-14T19:04:41.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:41 smithi102 ceph-mon[129280]: pgmap v324: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 311 B/s wr, 6 op/s 2023-07-14T19:04:41.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:41 smithi088 ceph-mon[128973]: pgmap v324: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 311 B/s wr, 6 op/s 2023-07-14T19:04:41.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:41 smithi078 ceph-mon[127425]: pgmap v324: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 311 B/s wr, 6 op/s 2023-07-14T19:04:41.848 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.845+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f691c02c8e0 con 0x7f691c100f50 2023-07-14T19:04:41.849 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.846+0000 7f6918ff9700 1 -- 172.21.15.78:0/330848790 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10808 (secure 0 0 0) 0x7f6910097680 con 0x7f691c100f50 2023-07-14T19:04:41.849 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:41.850 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:04:25.237091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24485":{"gid":24485,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:04:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f69040a1220 msgr2=0x7f69040a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f69040a1220 0x7f69040a36d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f690c005fd0 tx=0x7f690c005e40 comp rx=0 tx=0).stop 2023-07-14T19:04:41.851 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 msgr2=0x7f691c195cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 0x7f691c195cf0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f6910006010 tx=0x7f6910003b00 comp rx=0 tx=0).stop 2023-07-14T19:04:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 shutdown_connections 2023-07-14T19:04:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7f69040a1220 0x7f69040a36d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.852 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7f691c0fdee0 0x7f691c1957b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f691c1018e0 0x7f691c19cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 --2- 172.21.15.78:0/330848790 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7f691c100f50 0x7f691c195cf0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 >> 172.21.15.78:0/330848790 conn(0x7f691c0f98a0 msgr2=0x7f691c1043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 shutdown_connections 2023-07-14T19:04:41.853 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.848+0000 7f6921f3d700 1 -- 172.21.15.78:0/330848790 wait complete. 2023-07-14T19:04:41.854 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-07-14T19:04:41.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-07-14T19:04:41.986 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 -- 172.21.15.78:0/4163144205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fe6f0 msgr2=0x7fd5741082d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:41.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 --2- 172.21.15.78:0/4163144205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fe6f0 0x7fd5741082d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00b0a0 tx=0x7fd56c00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 -- 172.21.15.78:0/4163144205 shutdown_connections 2023-07-14T19:04:41.987 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 --2- 172.21.15.78:0/4163144205 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5741012f0 0x7fd574101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 --2- 172.21.15.78:0/4163144205 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fe6f0 0x7fd5741082d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 --2- 172.21.15.78:0/4163144205 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5740fdd60 0x7fd5740fe1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.988 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 -- 172.21.15.78:0/4163144205 >> 172.21.15.78:0/4163144205 conn(0x7fd5740f9990 msgr2=0x7fd5740fbda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 -- 172.21.15.78:0/4163144205 shutdown_connections 2023-07-14T19:04:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.981+0000 7fd57b912700 1 -- 172.21.15.78:0/4163144205 wait complete. 2023-07-14T19:04:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd57b912700 1 Processor -- start 2023-07-14T19:04:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd57b912700 1 -- start start 2023-07-14T19:04:41.989 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd57b912700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fdd60 0x7fd574195860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd57b912700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5740fe6f0 0x7fd574195da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd57b912700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5741012f0 0x7fd57419ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd57b912700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57419d3e0 con 0x7fd5740fdd60 2023-07-14T19:04:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd57b912700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57419d520 con 0x7fd5740fe6f0 2023-07-14T19:04:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd57b912700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57419d6b0 con 0x7fd5741012f0 2023-07-14T19:04:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd5796ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fdd60 0x7fd574195860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd5796ae700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fdd60 0x7fd574195860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36490/0 (socket says 172.21.15.78:36490) 2023-07-14T19:04:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd5796ae700 1 -- 172.21.15.78:0/2879752329 learned_addr learned my addr 172.21.15.78:0/2879752329 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.982+0000 7fd578ead700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5740fe6f0 0x7fd574195da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd5796ae700 1 -- 172.21.15.78:0/2879752329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5741012f0 msgr2=0x7fd57419ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd5796ae700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5741012f0 0x7fd57419ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd5796ae700 1 -- 172.21.15.78:0/2879752329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5740fe6f0 msgr2=0x7fd574195da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd5796ae700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5740fe6f0 0x7fd574195da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd5796ae700 1 -- 172.21.15.78:0/2879752329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c009d00 con 0x7fd5740fdd60 2023-07-14T19:04:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd5796ae700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fdd60 0x7fd574195860 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd57000d980 tx=0x7fd57000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd56a7fc700 1 -- 172.21.15.78:0/2879752329 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570004020 con 0x7fd5740fdd60 2023-07-14T19:04:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd56a7fc700 1 -- 172.21.15.78:0/2879752329 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fd570009730 con 0x7fd5740fdd60 2023-07-14T19:04:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57419d990 con 0x7fd5740fdd60 2023-07-14T19:04:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd56a7fc700 1 -- 172.21.15.78:0/2879752329 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd570003a00 con 0x7fd5740fdd60 2023-07-14T19:04:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.983+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57419de50 con 0x7fd5740fdd60 2023-07-14T19:04:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.984+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd57410b360 con 0x7fd5740fdd60 2023-07-14T19:04:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.986+0000 7fd56a7fc700 1 -- 172.21.15.78:0/2879752329 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fd570022020 con 0x7fd5740fdd60 2023-07-14T19:04:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.986+0000 7fd56a7fc700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd5600a1220 0x7fd5600a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.986+0000 7fd578ead700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd5600a1220 0x7fd5600a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.986+0000 7fd56a7fc700 1 -- 172.21.15.78:0/2879752329 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fd5700cc640 con 0x7fd5740fdd60 2023-07-14T19:04:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.987+0000 7fd578ead700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd5600a1220 0x7fd5600a36d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd56400ac50 tx=0x7fd56400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:41.990+0000 7fd56a7fc700 1 -- 172.21.15.78:0/2879752329 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fd5700958b0 con 0x7fd5740fdd60 2023-07-14T19:04:42.221 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.218+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd57419e6c0 con 0x7fd5740fdd60 2023-07-14T19:04:42.222 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.219+0000 7fd56a7fc700 1 -- 172.21.15.78:0/2879752329 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+10808 (secure 0 0 0) 0x7fd570098ed0 con 0x7fd5740fdd60 2023-07-14T19:04:42.223 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:42.223 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14559,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6837/4055842453","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6836","nonce":4055842453},{"type":"v1","addr":"172.21.15.78:6837","nonce":4055842453}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6839/680498429","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6838","nonce":680498429},{"type":"v1","addr":"172.21.15.78:6839","nonce":680498429}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6841/1070208","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6840","nonce":1070208},{"type":"v1","addr":"172.21.15.78:6841","nonce":1070208}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":12},{"gid":24418,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2462741496","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2462741496},{"type":"v1","addr":"172.21.15.102:6833","nonce":2462741496}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":24428,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.88:6833/244273805","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6832","nonce":244273805},{"type":"v1","addr":"172.21.15.88:6833","nonce":244273805}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":24440,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.78:6835/2843510081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6834","nonce":2843510081},{"type":"v1","addr":"172.21.15.78:6835","nonce":2843510081}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":4}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2023-07-14T19:02:30.340765+0000","modified":"2023-07-14T19:04:25.237091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24503,"mds_1":24487,"mds_2":24485},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24457":{"gid":24457,"name":"f","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6835/75062394","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":75062394},{"type":"v1","addr":"172.21.15.102:6835","nonce":75062394}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24464":{"gid":24464,"name":"e","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.88:6835/3667124043","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6834","nonce":3667124043},{"type":"v1","addr":"172.21.15.88:6835","nonce":3667124043}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24469":{"gid":24469,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.102:6837/3933565644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":3933565644},{"type":"v1","addr":"172.21.15.102:6837","nonce":3933565644}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24485":{"gid":24485,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":10,"addr":"172.21.15.88:6837/1950772288","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6836","nonce":1950772288},{"type":"v1","addr":"172.21.15.88:6837","nonce":1950772288}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24487":{"gid":24487,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":9,"addr":"172.21.15.102:6839/984471622","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":984471622},{"type":"v1","addr":"172.21.15.102:6839","nonce":984471622}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_24503":{"gid":24503,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.88:6839/1379790346","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6838","nonce":1379790346},{"type":"v1","addr":"172.21.15.88:6839","nonce":1379790346}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-07-14T19:04:42.224 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd5600a1220 msgr2=0x7fd5600a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd5600a1220 0x7fd5600a36d0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd56400ac50 tx=0x7fd56400a380 comp rx=0 tx=0).stop 2023-07-14T19:04:42.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fdd60 msgr2=0x7fd574195860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fdd60 0x7fd574195860 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fd57000d980 tx=0x7fd57000dc90 comp rx=0 tx=0).stop 2023-07-14T19:04:42.225 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 shutdown_connections 2023-07-14T19:04:42.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fd5600a1220 0x7fd5600a36d0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5741012f0 0x7fd57419ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fd5740fdd60 0x7fd574195860 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 --2- 172.21.15.78:0/2879752329 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fd5740fe6f0 0x7fd574195da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.226 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 >> 172.21.15.78:0/2879752329 conn(0x7fd5740f9990 msgr2=0x7fd5740fbad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:42.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 shutdown_connections 2023-07-14T19:04:42.227 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.221+0000 7fd57b912700 1 -- 172.21.15.78:0/2879752329 wait complete. 2023-07-14T19:04:42.227 INFO:teuthology.orchestra.run.smithi078.stderr:dumped fsmap epoch 24 2023-07-14T19:04:42.237 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-07-14T19:04:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 -- 172.21.15.78:0/2111614018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec0fe620 msgr2=0x7fcbec1081e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/2111614018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec0fe620 0x7fcbec1081e0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400b0a0 tx=0x7fcbe400b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 -- 172.21.15.78:0/2111614018 shutdown_connections 2023-07-14T19:04:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/2111614018 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec0fe620 0x7fcbec1081e0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/2111614018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbec101200 0x7fcbec101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/2111614018 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbec0fdc70 0x7fcbec0fe0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 -- 172.21.15.78:0/2111614018 >> 172.21.15.78:0/2111614018 conn(0x7fcbec0f98a0 msgr2=0x7fcbec0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 -- 172.21.15.78:0/2111614018 shutdown_connections 2023-07-14T19:04:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 -- 172.21.15.78:0/2111614018 wait complete. 2023-07-14T19:04:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 Processor -- start 2023-07-14T19:04:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.369+0000 7fcbf25c7700 1 -- start start 2023-07-14T19:04:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf25c7700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbec0fdc70 0x7fcbec100900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf25c7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbec0fe620 0x7fcbec0fef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf25c7700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec101200 0x7fcbec0ff480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf25c7700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0ffc20 con 0x7fcbec101200 2023-07-14T19:04:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf25c7700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec0ffd90 con 0x7fcbec0fdc70 2023-07-14T19:04:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf25c7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbec19f950 con 0x7fcbec0fe620 2023-07-14T19:04:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec101200 0x7fcbec0ff480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec101200 0x7fcbec0ff480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36508/0 (socket says 172.21.15.78:36508) 2023-07-14T19:04:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 -- 172.21.15.78:0/334197269 learned_addr learned my addr 172.21.15.78:0/334197269 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbeb7fe700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbec0fe620 0x7fcbec0fef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbebfff700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbec0fdc70 0x7fcbec100900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 -- 172.21.15.78:0/334197269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbec0fe620 msgr2=0x7fcbec0fef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbec0fe620 0x7fcbec0fef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 -- 172.21.15.78:0/334197269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbec0fdc70 msgr2=0x7fcbec100900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.385 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbec0fdc70 0x7fcbec100900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 -- 172.21.15.78:0/334197269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe4009d00 con 0x7fcbec101200 2023-07-14T19:04:42.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbebfff700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbec0fdc70 0x7fcbec100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:42.386 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf0b64700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec101200 0x7fcbec0ff480 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400b780 tx=0x7fcbe400bd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:42.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbe97fa700 1 -- 172.21.15.78:0/334197269 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe400e040 con 0x7fcbec101200 2023-07-14T19:04:42.387 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbe97fa700 1 -- 172.21.15.78:0/334197269 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fcbe4004e60 con 0x7fcbec101200 2023-07-14T19:04:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbec19fbd0 con 0x7fcbec101200 2023-07-14T19:04:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbe97fa700 1 -- 172.21.15.78:0/334197269 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe40052a0 con 0x7fcbec101200 2023-07-14T19:04:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.370+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbec1a0050 con 0x7fcbec101200 2023-07-14T19:04:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.371+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbec107340 con 0x7fcbec101200 2023-07-14T19:04:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.372+0000 7fcbe97fa700 1 -- 172.21.15.78:0/334197269 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fcbe401d470 con 0x7fcbec101200 2023-07-14T19:04:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.373+0000 7fcbe97fa700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcbd40a1220 0x7fcbd40a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.373+0000 7fcbebfff700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcbd40a1220 0x7fcbd40a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.373+0000 7fcbe97fa700 1 -- 172.21.15.78:0/334197269 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fcbe40cebf0 con 0x7fcbec101200 2023-07-14T19:04:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.374+0000 7fcbebfff700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcbd40a1220 0x7fcbd40a36d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0005fd0 tx=0x7fcbe0005f00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.377+0000 7fcbe97fa700 1 -- 172.21.15.78:0/334197269 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fcbe4097ef0 con 0x7fcbec101200 2023-07-14T19:04:42.546 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:42 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/330848790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:42.547 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:42 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2879752329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:42.573 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.570+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcbec02cb30 con 0x7fcbec101200 2023-07-14T19:04:42.575 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.572+0000 7fcbe97fa700 1 -- 172.21.15.78:0/334197269 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24640 (secure 0 0 0) 0x7fcbe40253b0 con 0x7fcbec101200 2023-07-14T19:04:42.575 INFO:teuthology.orchestra.run.smithi078.stdout: 2023-07-14T19:04:42.576 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":89,"fsid":"e2ca1a0e-2277-11ee-9b33-001a4aab830c","created":"2023-07-14T18:55:36.239751+0000","modified":"2023-07-14T19:04:31.284921+0000","last_up_change":"2023-07-14T19:01:49.714371+0000","last_in_change":"2023-07-14T19:01:31.961104+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-07-14T18:58:40.361042+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-07-14T19:02:28.269050+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-07-14T19:02:29.547771+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2023-07-14T19:02:32.797605+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"94b51247-9012-4bba-941b-2a524896715f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6802","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6803","nonce":1630210341}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6804","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6805","nonce":1630210341}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6808","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6809","nonce":1630210341}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6806","nonce":1630210341},{"type":"v1","addr":"172.21.15.78:6807","nonce":1630210341}]},"public_addr":"172.21.15.78:6803/1630210341","cluster_addr":"172.21.15.78:6805/1630210341","heartbeat_back_addr":"172.21.15.78:6809/1630210341","heartbeat_front_addr":"172.21.15.78:6807/1630210341","state":["exists","up"]},{"osd":1,"uuid":"e5f66a20-1101-4355-8ca4-9d3de118d5c8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6800","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6801","nonce":3959843234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6802","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6803","nonce":3959843234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6806","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6807","nonce":3959843234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6804","nonce":3959843234},{"type":"v1","addr":"172.21.15.88:6805","nonce":3959843234}]},"public_addr":"172.21.15.88:6801/3959843234","cluster_addr":"172.21.15.88:6803/3959843234","heartbeat_back_addr":"172.21.15.88:6807/3959843234","heartbeat_front_addr":"172.21.15.88:6805/3959843234","state":["exists","up"]},{"osd":2,"uuid":"180e6f44-2edd-427f-ac80-850d9e605f23","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6801","nonce":861816843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6803","nonce":861816843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6807","nonce":861816843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":861816843},{"type":"v1","addr":"172.21.15.102:6805","nonce":861816843}]},"public_addr":"172.21.15.102:6801/861816843","cluster_addr":"172.21.15.102:6803/861816843","heartbeat_back_addr":"172.21.15.102:6807/861816843","heartbeat_front_addr":"172.21.15.102:6805/861816843","state":["exists","up"]},{"osd":3,"uuid":"2e708d95-c299-4868-bbea-8ac0241d4118","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6810","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6811","nonce":4191618186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6812","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6813","nonce":4191618186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6816","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6817","nonce":4191618186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6814","nonce":4191618186},{"type":"v1","addr":"172.21.15.78:6815","nonce":4191618186}]},"public_addr":"172.21.15.78:6811/4191618186","cluster_addr":"172.21.15.78:6813/4191618186","heartbeat_back_addr":"172.21.15.78:6817/4191618186","heartbeat_front_addr":"172.21.15.78:6815/4191618186","state":["exists","up"]},{"osd":4,"uuid":"f2a8821e-94f2-4927-a1c6-c1bcce365fea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6808","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6809","nonce":1708042388}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6810","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6811","nonce":1708042388}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6814","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6815","nonce":1708042388}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6812","nonce":1708042388},{"type":"v1","addr":"172.21.15.88:6813","nonce":1708042388}]},"public_addr":"172.21.15.88:6809/1708042388","cluster_addr":"172.21.15.88:6811/1708042388","heartbeat_back_addr":"172.21.15.88:6815/1708042388","heartbeat_front_addr":"172.21.15.88:6813/1708042388","state":["exists","up"]},{"osd":5,"uuid":"7abc51f7-2397-4f2a-8046-0b93c4a56115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6809","nonce":3895937355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6811","nonce":3895937355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6815","nonce":3895937355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":3895937355},{"type":"v1","addr":"172.21.15.102:6813","nonce":3895937355}]},"public_addr":"172.21.15.102:6809/3895937355","cluster_addr":"172.21.15.102:6811/3895937355","heartbeat_back_addr":"172.21.15.102:6815/3895937355","heartbeat_front_addr":"172.21.15.102:6813/3895937355","state":["exists","up"]},{"osd":6,"uuid":"c02e47de-7c3a-41c9-bb4c-d339519b6a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6818","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6819","nonce":2536976719}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6820","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6821","nonce":2536976719}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6824","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6825","nonce":2536976719}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6822","nonce":2536976719},{"type":"v1","addr":"172.21.15.78:6823","nonce":2536976719}]},"public_addr":"172.21.15.78:6819/2536976719","cluster_addr":"172.21.15.78:6821/2536976719","heartbeat_back_addr":"172.21.15.78:6825/2536976719","heartbeat_front_addr":"172.21.15.78:6823/2536976719","state":["exists","up"]},{"osd":7,"uuid":"603c8010-0bcc-4cb4-9e64-308839b69ea1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":50,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6816","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6817","nonce":1000622113}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6818","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6819","nonce":1000622113}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6822","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6823","nonce":1000622113}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6820","nonce":1000622113},{"type":"v1","addr":"172.21.15.88:6821","nonce":1000622113}]},"public_addr":"172.21.15.88:6817/1000622113","cluster_addr":"172.21.15.88:6819/1000622113","heartbeat_back_addr":"172.21.15.88:6823/1000622113","heartbeat_front_addr":"172.21.15.88:6821/1000622113","state":["exists","up"]},{"osd":8,"uuid":"560f9ecc-e1d9-472d-af8b-ac1575902103","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":57,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6817","nonce":4258575075}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6819","nonce":4258575075}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6823","nonce":4258575075}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":4258575075},{"type":"v1","addr":"172.21.15.102:6821","nonce":4258575075}]},"public_addr":"172.21.15.102:6817/4258575075","cluster_addr":"172.21.15.102:6819/4258575075","heartbeat_back_addr":"172.21.15.102:6823/4258575075","heartbeat_front_addr":"172.21.15.102:6821/4258575075","state":["exists","up"]},{"osd":9,"uuid":"8afa084a-aebe-4fbf-af3b-36fcbf682dc4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":64,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6826","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6827","nonce":1815793298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6828","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6829","nonce":1815793298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6832","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6833","nonce":1815793298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.78:6830","nonce":1815793298},{"type":"v1","addr":"172.21.15.78:6831","nonce":1815793298}]},"public_addr":"172.21.15.78:6827/1815793298","cluster_addr":"172.21.15.78:6829/1815793298","heartbeat_back_addr":"172.21.15.78:6833/1815793298","heartbeat_front_addr":"172.21.15.78:6831/1815793298","state":["exists","up"]},{"osd":10,"uuid":"f150df7b-04f5-4103-bd4f-d93f48408f48","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":69,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6824","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6825","nonce":2764459881}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6826","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6827","nonce":2764459881}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6830","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6831","nonce":2764459881}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.88:6828","nonce":2764459881},{"type":"v1","addr":"172.21.15.88:6829","nonce":2764459881}]},"public_addr":"172.21.15.88:6825/2764459881","cluster_addr":"172.21.15.88:6827/2764459881","heartbeat_back_addr":"172.21.15.88:6831/2764459881","heartbeat_front_addr":"172.21.15.88:6829/2764459881","state":["exists","up"]},{"osd":11,"uuid":"dc848771-b0b4-4e08-838a-25775af1cefb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":76,"up_thru":86,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6825","nonce":383662404}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6827","nonce":383662404}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6831","nonce":383662404}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":383662404},{"type":"v1","addr":"172.21.15.102:6829","nonce":383662404}]},"public_addr":"172.21.15.102:6825/383662404","cluster_addr":"172.21.15.102:6827/383662404","heartbeat_back_addr":"172.21.15.102:6831/383662404","heartbeat_front_addr":"172.21.15.102:6829/383662404","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:57:54.817058+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:16.591011+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:37.341467+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:58:58.222081+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:19.664494+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T18:59:40.759330+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:01.850689+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:23.508538+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:00:44.242151+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:05.144582+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:27.107820+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-07-14T19:01:47.985985+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"2.1a","mappings":[{"from":8,"to":1}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.11","mappings":[{"from":4,"to":8}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"3.37","mappings":[{"from":5,"to":6}]},{"pgid":"4.f","mappings":[{"from":9,"to":10}]},{"pgid":"4.12","mappings":[{"from":5,"to":10}]},{"pgid":"4.16","mappings":[{"from":9,"to":8}]},{"pgid":"4.18","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.78:6800/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1800545696":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6800/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:6801/378103227":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/1797764467":"2023-07-15T18:56:30.036843+0000","172.21.15.78:6801/3249522727":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/21455120":"2023-07-15T18:56:30.036843+0000","172.21.15.78:0/3838414629":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/1459224377":"2023-07-15T18:56:08.189962+0000","172.21.15.78:0/2696766201":"2023-07-15T18:56:08.189962+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-07-14T19:04:42.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcbd40a1220 msgr2=0x7fcbd40a36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.577 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcbd40a1220 0x7fcbd40a36d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0005fd0 tx=0x7fcbe0005f00 comp rx=0 tx=0).stop 2023-07-14T19:04:42.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec101200 msgr2=0x7fcbec0ff480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec101200 0x7fcbec0ff480 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7fcbe400b780 tx=0x7fcbe400bd40 comp rx=0 tx=0).stop 2023-07-14T19:04:42.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 shutdown_connections 2023-07-14T19:04:42.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fcbd40a1220 0x7fcbd40a36d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.578 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fcbec0fdc70 0x7fcbec100900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fcbec101200 0x7fcbec0ff480 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 --2- 172.21.15.78:0/334197269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbec0fe620 0x7fcbec0fef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.579 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 >> 172.21.15.78:0/334197269 conn(0x7fcbec0f98a0 msgr2=0x7fcbec104480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:42.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 shutdown_connections 2023-07-14T19:04:42.580 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.574+0000 7fcbf25c7700 1 -- 172.21.15.78:0/334197269 wait complete. 2023-07-14T19:04:42.591 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-07-14T19:04:42.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/330848790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:42.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2879752329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:42.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/330848790' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:42.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2879752329' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2023-07-14T19:04:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.710+0000 7fb29e06b700 1 -- 172.21.15.78:0/4188085928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298107aa0 msgr2=0x7fb298109f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.710+0000 7fb29e06b700 1 --2- 172.21.15.78:0/4188085928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298107aa0 0x7fb298109f20 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7fb29000b0a0 tx=0x7fb29000b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.710+0000 7fb29e06b700 1 -- 172.21.15.78:0/4188085928 shutdown_connections 2023-07-14T19:04:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.710+0000 7fb29e06b700 1 --2- 172.21.15.78:0/4188085928 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb298074620 0x7fb298072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.710+0000 7fb29e06b700 1 --2- 172.21.15.78:0/4188085928 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298107aa0 0x7fb298109f20 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.710+0000 7fb29e06b700 1 --2- 172.21.15.78:0/4188085928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb298073040 0x7fb2980734b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.710+0000 7fb29e06b700 1 -- 172.21.15.78:0/4188085928 >> 172.21.15.78:0/4188085928 conn(0x7fb2980fb250 msgr2=0x7fb2980fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 -- 172.21.15.78:0/4188085928 shutdown_connections 2023-07-14T19:04:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 -- 172.21.15.78:0/4188085928 wait complete. 2023-07-14T19:04:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 Processor -- start 2023-07-14T19:04:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 -- start start 2023-07-14T19:04:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298073040 0x7fb298197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb298074620 0x7fb298198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb298107aa0 0x7fb29819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29819f6c0 con 0x7fb298073040 2023-07-14T19:04:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29819f800 con 0x7fb298107aa0 2023-07-14T19:04:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.711+0000 7fb29e06b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb29819f990 con 0x7fb298074620 2023-07-14T19:04:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298073040 0x7fb298197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298073040 0x7fb298197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36518/0 (socket says 172.21.15.78:36518) 2023-07-14T19:04:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 -- 172.21.15.78:0/1962440328 learned_addr learned my addr 172.21.15.78:0/1962440328 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb297fff700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb298107aa0 0x7fb29819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb296ffd700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb298074620 0x7fb298198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 -- 172.21.15.78:0/1962440328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb298074620 msgr2=0x7fb298198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb298074620 0x7fb298198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 -- 172.21.15.78:0/1962440328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb298107aa0 msgr2=0x7fb29819f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb298107aa0 0x7fb29819f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 -- 172.21.15.78:0/1962440328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb290009d00 con 0x7fb298073040 2023-07-14T19:04:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb297fff700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb298107aa0 0x7fb29819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb2977fe700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298073040 0x7fb298197b40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb28800e9a0 tx=0x7fb28800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb294ff9700 1 -- 172.21.15.78:0/1962440328 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb288009790 con 0x7fb298073040 2023-07-14T19:04:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb294ff9700 1 -- 172.21.15.78:0/1962440328 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb28800cbd0 con 0x7fb298073040 2023-07-14T19:04:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb29819fc70 con 0x7fb298073040 2023-07-14T19:04:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb294ff9700 1 -- 172.21.15.78:0/1962440328 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb288018680 con 0x7fb298073040 2023-07-14T19:04:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.712+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2981a0180 con 0x7fb298073040 2023-07-14T19:04:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.713+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2981061e0 con 0x7fb298073040 2023-07-14T19:04:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.714+0000 7fb294ff9700 1 -- 172.21.15.78:0/1962440328 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb28800cd40 con 0x7fb298073040 2023-07-14T19:04:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.715+0000 7fb294ff9700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb2800a11e0 0x7fb2800a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.715+0000 7fb296ffd700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb2800a11e0 0x7fb2800a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.715+0000 7fb294ff9700 1 -- 172.21.15.78:0/1962440328 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 9136+0+0 (secure 0 0 0) 0x7fb288014070 con 0x7fb298073040 2023-07-14T19:04:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.716+0000 7fb296ffd700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb2800a11e0 0x7fb2800a3690 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb28c005fd0 tx=0x7fb28c00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.719+0000 7fb294ff9700 1 -- 172.21.15.78:0/1962440328 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb2880965e0 con 0x7fb298073040 2023-07-14T19:04:42.953 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.950+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fb29802c8e0 con 0x7fb298073040 2023-07-14T19:04:42.959 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.956+0000 7fb294ff9700 1 -- 172.21.15.78:0/1962440328 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7fb288026090 con 0x7fb298073040 2023-07-14T19:04:42.961 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb2800a11e0 msgr2=0x7fb2800a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb2800a11e0 0x7fb2800a3690 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb28c005fd0 tx=0x7fb28c00a560 comp rx=0 tx=0).stop 2023-07-14T19:04:42.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298073040 msgr2=0x7fb298197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:42.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298073040 0x7fb298197b40 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7fb28800e9a0 tx=0x7fb28800ed60 comp rx=0 tx=0).stop 2023-07-14T19:04:42.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 shutdown_connections 2023-07-14T19:04:42.962 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb2800a11e0 0x7fb2800a3690 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb298107aa0 0x7fb29819f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb298073040 0x7fb298197b40 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 --2- 172.21.15.78:0/1962440328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb298074620 0x7fb298198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:42.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 >> 172.21.15.78:0/1962440328 conn(0x7fb2980fb250 msgr2=0x7fb2980fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:42.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 shutdown_connections 2023-07-14T19:04:42.964 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:42.958+0000 7fb29e06b700 1 -- 172.21.15.78:0/1962440328 wait complete. 2023-07-14T19:04:42.964 INFO:teuthology.orchestra.run.smithi078.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-07-14T19:04:42.974 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-07-14T19:04:43.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- 172.21.15.78:0/3099688935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc100750 msgr2=0x7ff4cc102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:43.103 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/3099688935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc100750 0x7ff4cc102b30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7ff4c0009990 tx=0x7ff4c0009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- 172.21.15.78:0/3099688935 shutdown_connections 2023-07-14T19:04:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/3099688935 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4cc105990 0x7ff4cc107df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/3099688935 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc100750 0x7ff4cc102b30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.104 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/3099688935 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4cc103070 0x7ff4cc105450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- 172.21.15.78:0/3099688935 >> 172.21.15.78:0/3099688935 conn(0x7ff4cc0fa1f0 msgr2=0x7ff4cc0fc620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:43.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- 172.21.15.78:0/3099688935 shutdown_connections 2023-07-14T19:04:43.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- 172.21.15.78:0/3099688935 wait complete. 2023-07-14T19:04:43.105 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 Processor -- start 2023-07-14T19:04:43.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- start start 2023-07-14T19:04:43.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4cc100750 0x7ff4cc1958c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:43.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4cc103070 0x7ff4cc195e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:43.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc105990 0x7ff4cc19ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:43.106 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc19d440 con 0x7ff4cc105990 2023-07-14T19:04:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc19d580 con 0x7ff4cc103070 2023-07-14T19:04:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.098+0000 7ff4d2f74700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc19d710 con 0x7ff4cc100750 2023-07-14T19:04:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4cc100750 0x7ff4cc1958c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d1511700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc105990 0x7ff4cc19ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:43.107 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4cc100750 0x7ff4cc1958c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.78:52890/0 (socket says 172.21.15.78:52890) 2023-07-14T19:04:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 -- 172.21.15.78:0/530141772 learned_addr learned my addr 172.21.15.78:0/530141772 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4cbfff700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4cc103070 0x7ff4cc195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 -- 172.21.15.78:0/530141772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4cc103070 msgr2=0x7ff4cc195e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4cc103070 0x7ff4cc195e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.108 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 -- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc105990 msgr2=0x7ff4cc19ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc105990 0x7ff4cc19ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 -- 172.21.15.78:0/530141772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4c00095f0 con 0x7ff4cc100750 2023-07-14T19:04:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d1511700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc105990 0x7ff4cc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-07-14T19:04:43.109 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d0d10700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4cc100750 0x7ff4cc1958c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00098e0 tx=0x7ff4c0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:43.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4c9ffb700 1 -- 172.21.15.78:0/530141772 <== mon.1 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c001d070 con 0x7ff4cc100750 2023-07-14T19:04:43.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc19d990 con 0x7ff4cc100750 2023-07-14T19:04:43.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4c9ffb700 1 -- 172.21.15.78:0/530141772 <== mon.1 v2:172.21.15.102:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7ff4c000be30 con 0x7ff4cc100750 2023-07-14T19:04:43.110 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.099+0000 7ff4c9ffb700 1 -- 172.21.15.78:0/530141772 <== mon.1 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0022db0 con 0x7ff4cc100750 2023-07-14T19:04:43.111 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.100+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc19ddb0 con 0x7ff4cc100750 2023-07-14T19:04:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.100+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc0fbd60 con 0x7ff4cc100750 2023-07-14T19:04:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.102+0000 7ff4c9ffb700 1 -- 172.21.15.78:0/530141772 <== mon.1 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7ff4c0022470 con 0x7ff4cc100750 2023-07-14T19:04:43.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.102+0000 7ff4c9ffb700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff4bc0a11e0 0x7ff4bc0a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.102+0000 7ff4cbfff700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff4bc0a11e0 0x7ff4bc0a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.102+0000 7ff4c9ffb700 1 -- 172.21.15.78:0/530141772 <== mon.1 v2:172.21.15.102:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9394+0+0 (secure 0 0 0) 0x7ff4c00ce950 con 0x7ff4cc100750 2023-07-14T19:04:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.103+0000 7ff4cbfff700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff4bc0a11e0 0x7ff4bc0a3690 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8007c60 tx=0x7ff4b80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:43.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.106+0000 7ff4c9ffb700 1 -- 172.21.15.78:0/530141772 <== mon.1 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7ff4c0097ae0 con 0x7ff4cc100750 2023-07-14T19:04:43.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-a[127401]: 2023-07-14T19:04:42.951+0000 7fe9f0e72700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:04:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.344+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff4cc02cd60 con 0x7ff4cc100750 2023-07-14T19:04:43.367 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b[154122]: ignoring --setuser ceph since I am not root 2023-07-14T19:04:43.368 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b[154122]: ignoring --setgroup ceph since I am not root 2023-07-14T19:04:43.369 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mds[154146]: -- 172.21.15.88:0/2011032447 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557ee469d1e0 con 0x557ee4672800 2023-07-14T19:04:43.369 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b[154122]: starting mds.b at 2023-07-14T19:04:43.370 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-e[153444]: ignoring --setuser ceph since I am not root 2023-07-14T19:04:43.370 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-e[153444]: ignoring --setgroup ceph since I am not root 2023-07-14T19:04:43.371 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-mds[153469]: -- 172.21.15.88:0/3643641349 <== mon.2 v2:172.21.15.88:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a765aff1e0 con 0x55a765ad4c00 2023-07-14T19:04:43.371 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-e[153444]: starting mds.e at 2023-07-14T19:04:43.372 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-h[153782]: ignoring --setuser ceph since I am not root 2023-07-14T19:04:43.372 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-h[153782]: ignoring --setgroup ceph since I am not root 2023-07-14T19:04:43.373 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:04:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-h[153782]: starting mds.h at 2023-07-14T19:04:43.374 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-i[155470]: ignoring --setuser ceph since I am not root 2023-07-14T19:04:43.374 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-i[155470]: ignoring --setgroup ceph since I am not root 2023-07-14T19:04:43.375 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-i[155470]: starting mds.i at 2023-07-14T19:04:43.375 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-c[155129]: ignoring --setuser ceph since I am not root 2023-07-14T19:04:43.376 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-c[155129]: ignoring --setgroup ceph since I am not root 2023-07-14T19:04:43.376 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-mds[155153]: -- 172.21.15.102:0/1208294553 <== mon.1 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557eebec31e0 con 0x557eebe98c00 2023-07-14T19:04:43.376 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-c[155129]: starting mds.c at 2023-07-14T19:04:43.377 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-f[154788]: ignoring --setuser ceph since I am not root 2023-07-14T19:04:43.378 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-f[154788]: ignoring --setgroup ceph since I am not root 2023-07-14T19:04:43.378 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-mds[154812]: -- 172.21.15.102:0/3973758148 <== mon.0 v2:172.21.15.78:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5634134191e0 con 0x5634133ee800 2023-07-14T19:04:43.378 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:04:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-f[154788]: starting mds.f at 2023-07-14T19:04:43.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 282 B/s wr, 7 op/s 2023-07-14T19:04:43.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/334197269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:43.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1962440328' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:04:43.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:04:43.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:04:43.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T19:04:43.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/1962440328' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:04:43.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:04:43.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:04:43.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/530141772' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:04:43.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:43 smithi102 ceph-mon[129280]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:04:43.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 282 B/s wr, 7 op/s 2023-07-14T19:04:43.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/334197269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:43.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1962440328' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:04:43.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:04:43.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:04:43.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T19:04:43.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/1962440328' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:04:43.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:04:43.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:04:43.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/530141772' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:04:43.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:43 smithi088 ceph-mon[128973]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:04:43.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: pgmap v325: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 282 B/s wr, 7 op/s 2023-07-14T19:04:43.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/334197269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-07-14T19:04:43.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1962440328' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2023-07-14T19:04:43.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2023-07-14T19:04:43.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-07-14T19:04:43.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: osdmap e90: 12 total, 12 up, 12 in 2023-07-14T19:04:43.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/1962440328' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2023-07-14T19:04:43.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: fsmap cephfs:0/3 6 up:standby, 3 failed 2023-07-14T19:04:43.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:04:43.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/530141772' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:04:43.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:43 smithi078 ceph-mon[127425]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2023-07-14T19:04:43.963 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.960+0000 7ff4c9ffb700 1 -- 172.21.15.78:0/530141772 <== mon.1 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff4cc02cd60 con 0x7ff4cc100750 2023-07-14T19:04:43.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff4bc0a11e0 msgr2=0x7ff4bc0a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:43.966 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff4bc0a11e0 0x7ff4bc0a3690 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff4b8007c60 tx=0x7ff4b80073d0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4cc100750 msgr2=0x7ff4cc1958c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:43.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4cc100750 0x7ff4cc1958c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff4c00098e0 tx=0x7ff4c0004730 comp rx=0 tx=0).stop 2023-07-14T19:04:43.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 shutdown_connections 2023-07-14T19:04:43.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7ff4bc0a11e0 0x7ff4bc0a3690 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.967 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7ff4cc105990 0x7ff4cc19ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4cc100750 0x7ff4cc1958c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 --2- 172.21.15.78:0/530141772 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7ff4cc103070 0x7ff4cc195e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:43.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 >> 172.21.15.78:0/530141772 conn(0x7ff4cc0fa1f0 msgr2=0x7ff4cc1013b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:43.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 shutdown_connections 2023-07-14T19:04:43.968 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:43.963+0000 7ff4d2f74700 1 -- 172.21.15.78:0/530141772 wait complete. 2023-07-14T19:04:43.982 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-07-14T19:04:44.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 -- 172.21.15.78:0/1303094280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 msgr2=0x7fca50106f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:44.112 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 --2- 172.21.15.78:0/1303094280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 0x7fca50106f60 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fca4800b0a0 tx=0x7fca4800b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 -- 172.21.15.78:0/1303094280 shutdown_connections 2023-07-14T19:04:44.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 --2- 172.21.15.78:0/1303094280 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 0x7fca50106f60 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 --2- 172.21.15.78:0/1303094280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca501000b0 0x7fca501004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 --2- 172.21.15.78:0/1303094280 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca500fed00 0x7fca500ff170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.113 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 -- 172.21.15.78:0/1303094280 >> 172.21.15.78:0/1303094280 conn(0x7fca500f98a0 msgr2=0x7fca500fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:44.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 -- 172.21.15.78:0/1303094280 shutdown_connections 2023-07-14T19:04:44.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.108+0000 7fca566cd700 1 -- 172.21.15.78:0/1303094280 wait complete. 2023-07-14T19:04:44.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca566cd700 1 Processor -- start 2023-07-14T19:04:44.114 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca566cd700 1 -- start start 2023-07-14T19:04:44.116 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca566cd700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca500fed00 0x7fca500fe760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:44.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca566cd700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 0x7fca5019a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:44.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca566cd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca501000b0 0x7fca5019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:44.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca566cd700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5019d4a0 con 0x7fca500ff6b0 2023-07-14T19:04:44.117 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca566cd700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5019d610 con 0x7fca500fed00 2023-07-14T19:04:44.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca566cd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca5019d7d0 con 0x7fca501000b0 2023-07-14T19:04:44.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 0x7fca5019a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:44.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4f7fe700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 0x7fca5019a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36548/0 (socket says 172.21.15.78:36548) 2023-07-14T19:04:44.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4f7fe700 1 -- 172.21.15.78:0/2100767939 learned_addr learned my addr 172.21.15.78:0/2100767939 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:44.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4ffff700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca500fed00 0x7fca500fe760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:44.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4f7fe700 1 -- 172.21.15.78:0/2100767939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca501000b0 msgr2=0x7fca5019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:44.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4f7fe700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca501000b0 0x7fca5019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4f7fe700 1 -- 172.21.15.78:0/2100767939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca500fed00 msgr2=0x7fca500fe760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:44.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4f7fe700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca500fed00 0x7fca500fe760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4f7fe700 1 -- 172.21.15.78:0/2100767939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca48009d00 con 0x7fca500ff6b0 2023-07-14T19:04:44.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.109+0000 7fca4ffff700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca500fed00 0x7fca500fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-07-14T19:04:44.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.110+0000 7fca4f7fe700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 0x7fca5019a6c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fca4400c920 tx=0x7fca4400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:44.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.110+0000 7fca4d7fa700 1 -- 172.21.15.78:0/2100767939 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44007b00 con 0x7fca500ff6b0 2023-07-14T19:04:44.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.110+0000 7fca4d7fa700 1 -- 172.21.15.78:0/2100767939 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fca44004cc0 con 0x7fca500ff6b0 2023-07-14T19:04:44.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.110+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca5019dab0 con 0x7fca500ff6b0 2023-07-14T19:04:44.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.110+0000 7fca4d7fa700 1 -- 172.21.15.78:0/2100767939 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca44005590 con 0x7fca500ff6b0 2023-07-14T19:04:44.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.110+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca5019dfc0 con 0x7fca500ff6b0 2023-07-14T19:04:44.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.111+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca5010a160 con 0x7fca500ff6b0 2023-07-14T19:04:44.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.112+0000 7fca4d7fa700 1 -- 172.21.15.78:0/2100767939 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fca44004750 con 0x7fca500ff6b0 2023-07-14T19:04:44.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.112+0000 7fca4d7fa700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fca380a11e0 0x7fca380a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:44.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.112+0000 7fca4ffff700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fca380a11e0 0x7fca380a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:44.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.112+0000 7fca4d7fa700 1 -- 172.21.15.78:0/2100767939 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9394+0+0 (secure 0 0 0) 0x7fca440ccef0 con 0x7fca500ff6b0 2023-07-14T19:04:44.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.113+0000 7fca4ffff700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fca380a11e0 0x7fca380a3690 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fca40005fd0 tx=0x7fca40005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:44.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.116+0000 7fca4d7fa700 1 -- 172.21.15.78:0/2100767939 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fca440f9780 con 0x7fca500ff6b0 2023-07-14T19:04:44.308 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.304+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fca50105ab0 con 0x7fca500ff6b0 2023-07-14T19:04:44.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:04:44.630 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:04:44.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: Cluster is now healthy 2023-07-14T19:04:44.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.102:6835/1009350452,v1:172.21.15.102:6838/1009350452] up:boot 2023-07-14T19:04:44.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.102:6837/3173685693,v1:172.21.15.102:6839/3173685693] up:boot 2023-07-14T19:04:44.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6834/922232416,v1:172.21.15.88:6835/922232416] up:boot 2023-07-14T19:04:44.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.102:6834/2462183214,v1:172.21.15.102:6836/2462183214] up:boot 2023-07-14T19:04:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6836/1600164562,v1:172.21.15.88:6837/1600164562] up:boot 2023-07-14T19:04:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: mds.? [v2:172.21.15.88:6838/3326216893,v1:172.21.15.88:6839/3326216893] up:boot 2023-07-14T19:04:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:04:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: fsmap 12 up:standby 2023-07-14T19:04:44.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:04:44.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:04:44.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:04:44.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:04:44.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:04:44.633 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:04:44.634 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:44 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2100767939' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:44.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:04:44.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:04:44.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: Cluster is now healthy 2023-07-14T19:04:44.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.102:6835/1009350452,v1:172.21.15.102:6838/1009350452] up:boot 2023-07-14T19:04:44.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.102:6837/3173685693,v1:172.21.15.102:6839/3173685693] up:boot 2023-07-14T19:04:44.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6834/922232416,v1:172.21.15.88:6835/922232416] up:boot 2023-07-14T19:04:44.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.102:6834/2462183214,v1:172.21.15.102:6836/2462183214] up:boot 2023-07-14T19:04:44.713 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6836/1600164562,v1:172.21.15.88:6837/1600164562] up:boot 2023-07-14T19:04:44.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: mds.? [v2:172.21.15.88:6838/3326216893,v1:172.21.15.88:6839/3326216893] up:boot 2023-07-14T19:04:44.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:04:44.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: fsmap 12 up:standby 2023-07-14T19:04:44.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:04:44.714 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:04:44.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:04:44.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:04:44.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:04:44.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:04:44.715 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:44 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2100767939' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2023-07-14T19:04:44.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2023-07-14T19:04:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: Cluster is now healthy 2023-07-14T19:04:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.102:6835/1009350452,v1:172.21.15.102:6838/1009350452] up:boot 2023-07-14T19:04:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.102:6837/3173685693,v1:172.21.15.102:6839/3173685693] up:boot 2023-07-14T19:04:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6834/922232416,v1:172.21.15.88:6835/922232416] up:boot 2023-07-14T19:04:44.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.102:6834/2462183214,v1:172.21.15.102:6836/2462183214] up:boot 2023-07-14T19:04:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6836/1600164562,v1:172.21.15.88:6837/1600164562] up:boot 2023-07-14T19:04:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: mds.? [v2:172.21.15.88:6838/3326216893,v1:172.21.15.88:6839/3326216893] up:boot 2023-07-14T19:04:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2023-07-14T19:04:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: fsmap 12 up:standby 2023-07-14T19:04:44.799 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2023-07-14T19:04:44.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2023-07-14T19:04:44.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2023-07-14T19:04:44.800 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2023-07-14T19:04:44.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2023-07-14T19:04:44.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: from='mgr.14152 172.21.15.78:0/3646937739' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2023-07-14T19:04:44.801 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:44 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2100767939' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:44.969 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.965+0000 7fca4d7fa700 1 -- 172.21.15.78:0/2100767939 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v91) v1 ==== 181+0+0 (secure 0 0 0) 0x7fca4401d020 con 0x7fca500ff6b0 2023-07-14T19:04:44.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fca380a11e0 msgr2=0x7fca380a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:44.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fca380a11e0 0x7fca380a3690 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fca40005fd0 tx=0x7fca40005ee0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.972 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 msgr2=0x7fca5019a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:44.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 0x7fca5019a6c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fca4400c920 tx=0x7fca4400cce0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 shutdown_connections 2023-07-14T19:04:44.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fca380a11e0 0x7fca380a3690 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.973 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fca500fed00 0x7fca500fe760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fca501000b0 0x7fca5019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 --2- 172.21.15.78:0/2100767939 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fca500ff6b0 0x7fca5019a6c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:44.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 >> 172.21.15.78:0/2100767939 conn(0x7fca500f98a0 msgr2=0x7fca500fbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:44.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 shutdown_connections 2023-07-14T19:04:44.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:44.968+0000 7fca566cd700 1 -- 172.21.15.78:0/2100767939 wait complete. 2023-07-14T19:04:44.975 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_metadata' removed 2023-07-14T19:04:44.985 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-07-14T19:04:45.118 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/1990761251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e4100690 msgr2=0x7fe6e4106f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:45.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/1990761251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e4100690 0x7fe6e4106f50 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00b0a0 tx=0x7fe6dc00b3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/1990761251 shutdown_connections 2023-07-14T19:04:45.119 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/1990761251 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e4100690 0x7fe6e4106f50 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/1990761251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6e40feb10 0x7fe6e40fef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.120 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/1990761251 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e40ffd00 0x7fe6e4100150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/1990761251 >> 172.21.15.78:0/1990761251 conn(0x7fe6e40fa130 msgr2=0x7fe6e40fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:45.121 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/1990761251 shutdown_connections 2023-07-14T19:04:45.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/1990761251 wait complete. 2023-07-14T19:04:45.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 Processor -- start 2023-07-14T19:04:45.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 -- start start 2023-07-14T19:04:45.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e40feb10 0x7fe6e4068330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:45.122 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e40ffd00 0x7fe6e419a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:45.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.109+0000 7fe6eb8ab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6e4100690 0x7fe6e419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:45.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6eb8ab700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e419d400 con 0x7fe6e40ffd00 2023-07-14T19:04:45.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6eb8ab700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e419d570 con 0x7fe6e40feb10 2023-07-14T19:04:45.123 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6eb8ab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e419d730 con 0x7fe6e4100690 2023-07-14T19:04:45.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e40ffd00 0x7fe6e419a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:45.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e40ffd00 0x7fe6e419a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36560/0 (socket says 172.21.15.78:36560) 2023-07-14T19:04:45.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 -- 172.21.15.78:0/2498777346 learned_addr learned my addr 172.21.15.78:0/2498777346 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:45.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e9647700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e40feb10 0x7fe6e4068330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:45.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e9e48700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6e4100690 0x7fe6e419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:45.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 -- 172.21.15.78:0/2498777346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6e4100690 msgr2=0x7fe6e419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:45.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6e4100690 0x7fe6e419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 -- 172.21.15.78:0/2498777346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e40feb10 msgr2=0x7fe6e4068330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:45.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e40feb10 0x7fe6e4068330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 -- 172.21.15.78:0/2498777346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc009d00 con 0x7fe6e40ffd00 2023-07-14T19:04:45.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6e8e46700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e40ffd00 0x7fe6e419a5e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000c9b0 tx=0x7fe6e000ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:45.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6da7fc700 1 -- 172.21.15.78:0/2498777346 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0007b00 con 0x7fe6e40ffd00 2023-07-14T19:04:45.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6da7fc700 1 -- 172.21.15.78:0/2498777346 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fe6e000ce50 con 0x7fe6e40ffd00 2023-07-14T19:04:45.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e419da10 con 0x7fe6e40ffd00 2023-07-14T19:04:45.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6da7fc700 1 -- 172.21.15.78:0/2498777346 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e0018420 con 0x7fe6e40ffd00 2023-07-14T19:04:45.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.110+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e419df20 con 0x7fe6e40ffd00 2023-07-14T19:04:45.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.111+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e410a1f0 con 0x7fe6e40ffd00 2023-07-14T19:04:45.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.112+0000 7fe6da7fc700 1 -- 172.21.15.78:0/2498777346 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fe6e000f450 con 0x7fe6e40ffd00 2023-07-14T19:04:45.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.113+0000 7fe6da7fc700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe6d00a11e0 0x7fe6d00a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:45.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.113+0000 7fe6e9647700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe6d00a11e0 0x7fe6d00a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:45.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.113+0000 7fe6da7fc700 1 -- 172.21.15.78:0/2498777346 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8840+0+0 (secure 0 0 0) 0x7fe6e00cced0 con 0x7fe6e40ffd00 2023-07-14T19:04:45.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.114+0000 7fe6e9647700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe6d00a11e0 0x7fe6d00a3690 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe6e40ffb60 tx=0x7fe6d400a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:45.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.117+0000 7fe6da7fc700 1 -- 172.21.15.78:0/2498777346 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fe6e00962f0 con 0x7fe6e40ffd00 2023-07-14T19:04:45.311 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.308+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe6e41058f0 con 0x7fe6e40ffd00 2023-07-14T19:04:45.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:45 smithi102 ceph-mon[129280]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-07-14T19:04:45.631 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:45 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2100767939' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:04:45.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:45 smithi102 ceph-mon[129280]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T19:04:45.632 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:45 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2498777346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:45.711 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:45 smithi088 ceph-mon[128973]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-07-14T19:04:45.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:45 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2100767939' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:04:45.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:45 smithi088 ceph-mon[128973]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T19:04:45.712 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:45 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2498777346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:45.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:45 smithi078 ceph-mon[127425]: pgmap v327: 129 pgs: 129 active+clean; 644 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 307 B/s wr, 6 op/s 2023-07-14T19:04:45.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:45 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2100767939' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:04:45.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:45 smithi078 ceph-mon[127425]: osdmap e91: 12 total, 12 up, 12 in 2023-07-14T19:04:45.798 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:45 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2498777346' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:45.971 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.967+0000 7fe6da7fc700 1 -- 172.21.15.78:0/2498777346 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v92) v1 ==== 169+0+0 (secure 0 0 0) 0x7fe6e000f700 con 0x7fe6e40ffd00 2023-07-14T19:04:45.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe6d00a11e0 msgr2=0x7fe6d00a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:45.974 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe6d00a11e0 0x7fe6d00a3690 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe6e40ffb60 tx=0x7fe6d400a3b0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e40ffd00 msgr2=0x7fe6e419a5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:45.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e40ffd00 0x7fe6e419a5e0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fe6e000c9b0 tx=0x7fe6e000ccc0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.975 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 shutdown_connections 2023-07-14T19:04:45.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fe6d00a11e0 0x7fe6d00a3690 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fe6e40feb10 0x7fe6e4068330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe6e4100690 0x7fe6e419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 --2- 172.21.15.78:0/2498777346 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fe6e40ffd00 0x7fe6e419a5e0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:45.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 >> 172.21.15.78:0/2498777346 conn(0x7fe6e40fa130 msgr2=0x7fe6e40fc280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:45.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 shutdown_connections 2023-07-14T19:04:45.977 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:45.971+0000 7fe6eb8ab700 1 -- 172.21.15.78:0/2498777346 wait complete. 2023-07-14T19:04:45.978 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data' removed 2023-07-14T19:04:45.990 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2023-07-14T19:04:46.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 -- 172.21.15.78:0/29302487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 msgr2=0x7fb8bc0fe2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:46.124 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 --2- 172.21.15.78:0/29302487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 0x7fb8bc0fe2f0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac009990 tx=0x7fb8ac009ca0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 -- 172.21.15.78:0/29302487 shutdown_connections 2023-07-14T19:04:46.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 --2- 172.21.15.78:0/29302487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8bc1018e0 0x7fb8bc1081a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 --2- 172.21.15.78:0/29302487 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 0x7fb8bc0fe2f0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.125 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 --2- 172.21.15.78:0/29302487 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8bc100f50 0x7fb8bc1013a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 -- 172.21.15.78:0/29302487 >> 172.21.15.78:0/29302487 conn(0x7fb8bc0f98a0 msgr2=0x7fb8bc0fbcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 -- 172.21.15.78:0/29302487 shutdown_connections 2023-07-14T19:04:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.117+0000 7fb8c3594700 1 -- 172.21.15.78:0/29302487 wait complete. 2023-07-14T19:04:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c3594700 1 Processor -- start 2023-07-14T19:04:46.126 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c3594700 1 -- start start 2023-07-14T19:04:46.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c3594700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 0x7fb8bc1008c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:46.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c3594700 1 --2- >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8bc100f50 0x7fb8bc0fef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:46.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c3594700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8bc1018e0 0x7fb8bc0ff440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:46.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c3594700 1 -- --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0ffbe0 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.127 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c3594700 1 -- --> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0ffd50 con 0x7fb8bc100f50 2023-07-14T19:04:46.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c3594700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc19f920 con 0x7fb8bc1018e0 2023-07-14T19:04:46.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 0x7fb8bc1008c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:46.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 --2- >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 0x7fb8bc1008c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.78:3300/0 says I am v2:172.21.15.78:36570/0 (socket says 172.21.15.78:36570) 2023-07-14T19:04:46.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 -- 172.21.15.78:0/3381637241 learned_addr learned my addr 172.21.15.78:0/3381637241 (peer_addr_for_me v2:172.21.15.78:0/0) 2023-07-14T19:04:46.128 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c0b2f700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8bc100f50 0x7fb8bc0fef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:46.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1b31700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8bc1018e0 0x7fb8bc0ff440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:46.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 -- 172.21.15.78:0/3381637241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8bc1018e0 msgr2=0x7fb8bc0ff440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:46.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8bc1018e0 0x7fb8bc0ff440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 -- 172.21.15.78:0/3381637241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8bc100f50 msgr2=0x7fb8bc0fef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:46.129 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8bc100f50 0x7fb8bc0fef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 -- 172.21.15.78:0/3381637241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac0095f0 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c0b2f700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8bc100f50 0x7fb8bc0fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:46.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1b31700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8bc1018e0 0x7fb8bc0ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-07-14T19:04:46.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.118+0000 7fb8c1330700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 0x7fb8bc1008c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c010 tx=0x7fb8ac005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:46.130 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.119+0000 7fb8b27fc700 1 -- 172.21.15.78:0/3381637241 <== mon.0 v2:172.21.15.78:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac01d070 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.119+0000 7fb8b27fc700 1 -- 172.21.15.78:0/3381637241 <== mon.0 v2:172.21.15.78:3300/0 2 ==== config(25 keys) v1 ==== 1067+0+0 (secure 0 0 0) 0x7fb8ac004100 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.119+0000 7fb8b27fc700 1 -- 172.21.15.78:0/3381637241 <== mon.0 v2:172.21.15.78:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac005290 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.119+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc19fba0 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.119+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc19ff70 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.131 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.120+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc1073f0 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.132 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.121+0000 7fb8b27fc700 1 -- 172.21.15.78:0/3381637241 <== mon.0 v2:172.21.15.78:3300/0 4 ==== mgrmap(e 19) v1 ==== 135766+0+0 (secure 0 0 0) 0x7fb8ac0047e0 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.121+0000 7fb8b27fc700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb8a80a11e0 0x7fb8a80a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-07-14T19:04:46.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.121+0000 7fb8c0b2f700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb8a80a11e0 0x7fb8a80a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-07-14T19:04:46.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.122+0000 7fb8b27fc700 1 -- 172.21.15.78:0/3381637241 <== mon.0 v2:172.21.15.78:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8252+0+0 (secure 0 0 0) 0x7fb8ac0cdfb0 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.133 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.122+0000 7fb8c0b2f700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb8a80a11e0 0x7fb8a80a3690 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8005950 tx=0x7fb8b8005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-07-14T19:04:46.134 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.125+0000 7fb8b27fc700 1 -- 172.21.15.78:0/3381637241 <== mon.0 v2:172.21.15.78:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+178025 (secure 0 0 0) 0x7fb8ac097620 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.316 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.313+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 --> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb8bc02cd60 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.976 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.973+0000 7fb8b27fc700 1 -- 172.21.15.78:0/3381637241 <== mon.0 v2:172.21.15.78:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v93) v1 ==== 178+0+0 (secure 0 0 0) 0x7fb8bc02cd60 con 0x7fb8bc0fdee0 2023-07-14T19:04:46.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb8a80a11e0 msgr2=0x7fb8a80a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:46.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb8a80a11e0 0x7fb8a80a3690 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb8b8005950 tx=0x7fb8b8005f50 comp rx=0 tx=0).stop 2023-07-14T19:04:46.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 msgr2=0x7fb8bc1008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-07-14T19:04:46.981 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 0x7fb8bc1008c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c010 tx=0x7fb8ac005ef0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 shutdown_connections 2023-07-14T19:04:46.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:6800/2266091126,v1:172.21.15.78:6801/2266091126] conn(0x7fb8a80a11e0 0x7fb8a80a3690 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb8bc1018e0 0x7fb8bc0ff440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.78:3300/0,v1:172.21.15.78:6789/0] conn(0x7fb8bc0fdee0 0x7fb8bc1008c0 secure :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac00c010 tx=0x7fb8ac005ef0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.982 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 --2- 172.21.15.78:0/3381637241 >> [v2:172.21.15.88:3300/0,v1:172.21.15.88:6789/0] conn(0x7fb8bc100f50 0x7fb8bc0fef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-07-14T19:04:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.977+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 >> 172.21.15.78:0/3381637241 conn(0x7fb8bc0f98a0 msgr2=0x7fb8bc104530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-07-14T19:04:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.978+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 shutdown_connections 2023-07-14T19:04:46.983 INFO:teuthology.orchestra.run.smithi078.stderr:2023-07-14T19:04:46.978+0000 7fb8c3594700 1 -- 172.21.15.78:0/3381637241 wait complete. 2023-07-14T19:04:46.984 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'cephfs_data_ec' removed 2023-07-14T19:04:46.995 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:04:47.022 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:04:47.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:04:47.099 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-07-14T19:04:47.099 DEBUG:teuthology.orchestra.run.smithi078:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:04:47.117 DEBUG:teuthology.orchestra.run.smithi088:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:04:47.135 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-07-14T19:04:47.159 INFO:tasks.cephadm:Stopping all daemons... 2023-07-14T19:04:47.159 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-07-14T19:04:47.159 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.a 2023-07-14T19:04:47.296 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:46 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/2498777346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:04:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:46 smithi078 ceph-mon[127425]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T19:04:47.297 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:46 smithi078 ceph-mon[127425]: pgmap v330: 33 pgs: 33 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:04:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:46 smithi078 ceph-mon[127425]: from='client.? 172.21.15.78:0/3381637241' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:47.298 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:47 smithi078 systemd[1]: Stopping Ceph mon.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:04:47.380 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:46 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/2498777346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:04:47.380 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:46 smithi102 ceph-mon[129280]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T19:04:47.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:46 smithi102 ceph-mon[129280]: pgmap v330: 33 pgs: 33 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:04:47.381 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:46 smithi102 ceph-mon[129280]: from='client.? 172.21.15.78:0/3381637241' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:47.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:46 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/2498777346' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2023-07-14T19:04:47.461 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:46 smithi088 ceph-mon[128973]: osdmap e92: 12 total, 12 up, 12 in 2023-07-14T19:04:47.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:46 smithi088 ceph-mon[128973]: pgmap v330: 33 pgs: 33 active+clean; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2023-07-14T19:04:47.462 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:46 smithi088 ceph-mon[128973]: from='client.? 172.21.15.78:0/3381637241' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2023-07-14T19:04:47.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-a[127401]: 2023-07-14T19:04:47.508+0000 7fe9f667d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:04:47.797 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-a[127401]: 2023-07-14T19:04:47.508+0000 7fe9f667d700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-07-14T19:04:48.134 INFO:journalctl@ceph.mon.a.smithi078.stdout:Jul 14 19:04:47 smithi078 bash[180184]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-a 2023-07-14T19:04:48.141 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.a.service' 2023-07-14T19:04:48.194 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:04:48.195 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-07-14T19:04:48.195 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-07-14T19:04:48.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.b 2023-07-14T19:04:48.589 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:48 smithi088 systemd[1]: Stopping Ceph mon.b for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:04:48.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-b[128949]: 2023-07-14T19:04:48.586+0000 7fc3b1342700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:04:48.961 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-b[128949]: 2023-07-14T19:04:48.586+0000 7fc3b1342700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-07-14T19:04:49.398 INFO:journalctl@ceph.mon.b.smithi088.stdout:Jul 14 19:04:49 smithi088 bash[156230]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-b 2023-07-14T19:04:49.404 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.b.service' 2023-07-14T19:04:49.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:04:49.458 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-07-14T19:04:49.459 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-07-14T19:04:49.460 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.c 2023-07-14T19:04:49.832 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:49 smithi102 systemd[1]: Stopping Ceph mon.c for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:04:50.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-c[129256]: 2023-07-14T19:04:49.827+0000 7f8855ab4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:04:50.130 INFO:journalctl@ceph.mon.c.smithi102.stdout:Jul 14 19:04:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mon-c[129256]: 2023-07-14T19:04:49.827+0000 7f8855ab4700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-07-14T19:04:50.361 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mon.c.service' 2023-07-14T19:04:50.414 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:04:50.415 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-07-14T19:04:50.415 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-07-14T19:04:50.416 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.x 2023-07-14T19:04:50.779 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:50 smithi078 systemd[1]: Stopping Ceph mgr.x for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:04:50.780 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:04:50.775+0000 7ff350c2f700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:04:50.780 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x[127670]: 2023-07-14T19:04:50.775+0000 7ff350c2f700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2023-07-14T19:04:51.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.x.service' 2023-07-14T19:04:51.334 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:51 smithi078 bash[180322]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-x 2023-07-14T19:04:51.334 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:51 smithi078 systemd[1]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.x.service: Succeeded. 2023-07-14T19:04:51.335 INFO:journalctl@ceph.mgr.x.smithi078.stdout:Jul 14 19:04:51 smithi078 systemd[1]: Stopped Ceph mgr.x for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:04:51.345 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:04:51.346 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-07-14T19:04:51.346 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2023-07-14T19:04:51.347 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.y 2023-07-14T19:04:51.711 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 19:04:51 smithi088 systemd[1]: Stopping Ceph mgr.y for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:04:52.056 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 19:04:52 smithi088 bash[156397]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-y 2023-07-14T19:04:52.234 INFO:journalctl@ceph.mgr.y.smithi088.stdout:Jul 14 19:04:52 smithi088 systemd[1]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.y.service: Main process exited, code=exited, status=143/n/a 2023-07-14T19:04:52.237 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.y.service' 2023-07-14T19:04:52.275 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:04:52.275 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2023-07-14T19:04:52.276 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2023-07-14T19:04:52.277 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.z 2023-07-14T19:04:52.630 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 19:04:52 smithi102 systemd[1]: Stopping Ceph mgr.z for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:04:52.985 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 19:04:52 smithi102 bash[157736]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mgr-z 2023-07-14T19:04:52.985 INFO:journalctl@ceph.mgr.z.smithi102.stdout:Jul 14 19:04:52 smithi102 systemd[1]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.z.service: Main process exited, code=exited, status=143/n/a 2023-07-14T19:04:53.166 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mgr.z.service' 2023-07-14T19:04:53.202 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:04:53.202 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2023-07-14T19:04:53.203 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-07-14T19:04:53.203 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.0 2023-07-14T19:04:53.546 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 19:04:53 smithi078 systemd[1]: Stopping Ceph osd.0 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:04:54.046 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 19:04:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-0[140506]: 2023-07-14T19:04:53.556+0000 7f1fdf689700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:04:54.047 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 19:04:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-0[140506]: 2023-07-14T19:04:53.556+0000 7f1fdf689700 -1 osd.0 93 *** Got signal Terminated *** 2023-07-14T19:04:54.047 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 19:04:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-0[140506]: 2023-07-14T19:04:53.556+0000 7f1fdf689700 -1 osd.0 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:04:59.297 INFO:journalctl@ceph.osd.0.smithi078.stdout:Jul 14 19:04:59 smithi078 bash[180467]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-0 2023-07-14T19:05:00.593 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.0.service' 2023-07-14T19:05:00.632 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:05:00.633 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-07-14T19:05:00.633 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-07-14T19:05:00.634 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.1 2023-07-14T19:05:00.961 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 19:05:00 smithi088 systemd[1]: Stopping Ceph osd.1 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:05:01.461 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 19:05:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-1[134087]: 2023-07-14T19:05:00.985+0000 7fa918f33700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:05:01.462 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 19:05:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-1[134087]: 2023-07-14T19:05:00.985+0000 7fa918f33700 -1 osd.1 93 *** Got signal Terminated *** 2023-07-14T19:05:01.462 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 19:05:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-1[134087]: 2023-07-14T19:05:00.985+0000 7fa918f33700 -1 osd.1 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:05:06.769 INFO:journalctl@ceph.osd.1.smithi088.stdout:Jul 14 19:05:06 smithi088 bash[156546]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-1 2023-07-14T19:05:08.039 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.1.service' 2023-07-14T19:05:08.091 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:05:08.092 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-07-14T19:05:08.092 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-07-14T19:05:08.092 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.2 2023-07-14T19:05:08.380 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 19:05:08 smithi102 systemd[1]: Stopping Ceph osd.2 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:05:08.880 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 19:05:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-2[135106]: 2023-07-14T19:05:08.570+0000 7f014ab7b700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:05:08.880 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 19:05:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-2[135106]: 2023-07-14T19:05:08.570+0000 7f014ab7b700 -1 osd.2 93 *** Got signal Terminated *** 2023-07-14T19:05:08.880 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 19:05:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-2[135106]: 2023-07-14T19:05:08.570+0000 7f014ab7b700 -1 osd.2 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:05:14.435 INFO:journalctl@ceph.osd.2.smithi102.stdout:Jul 14 19:05:14 smithi102 bash[157883]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-2 2023-07-14T19:05:15.698 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.2.service' 2023-07-14T19:05:15.739 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:05:15.739 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-07-14T19:05:15.740 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-07-14T19:05:15.740 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.3 2023-07-14T19:05:16.047 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 14 19:05:15 smithi078 systemd[1]: Stopping Ceph osd.3 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:05:16.546 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 14 19:05:16 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-3[145928]: 2023-07-14T19:05:16.096+0000 7f0ee25b1700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:05:16.547 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 14 19:05:16 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-3[145928]: 2023-07-14T19:05:16.096+0000 7f0ee25b1700 -1 osd.3 93 *** Got signal Terminated *** 2023-07-14T19:05:16.547 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 14 19:05:16 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-3[145928]: 2023-07-14T19:05:16.096+0000 7f0ee25b1700 -1 osd.3 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:05:20.130 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:19 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:19.702+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:20.747 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:20 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:20.431+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:21.130 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:20 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:20.743+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:21.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:20 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:20.941+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:21.742 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:21 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:21.469+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:21.783 INFO:journalctl@ceph.osd.3.smithi078.stdout:Jul 14 19:05:21 smithi078 bash[180751]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-3 2023-07-14T19:05:22.130 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:21 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:21.739+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:22.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:21 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:21.956+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:22.755 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:22 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:22.469+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:23.038 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:22 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:22.917+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:23.045 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.3.service' 2023-07-14T19:05:23.086 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:05:23.087 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-07-14T19:05:23.088 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-07-14T19:05:23.088 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.4 2023-07-14T19:05:23.119 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:22 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:22.820+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:23.130 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:22 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:22.751+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:23.449 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 19:05:23 smithi088 systemd[1]: Stopping Ceph osd.4 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:05:23.711 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 19:05:23 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-4[139487]: 2023-07-14T19:05:23.445+0000 7f303ece7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:05:23.711 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 19:05:23 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-4[139487]: 2023-07-14T19:05:23.445+0000 7f303ece7700 -1 osd.4 93 *** Got signal Terminated *** 2023-07-14T19:05:23.712 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 19:05:23 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-4[139487]: 2023-07-14T19:05:23.445+0000 7f303ece7700 -1 osd.4 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:05:23.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:23 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:23.722+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:23.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:23 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:23.500+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:24.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:23 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:23.805+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:24.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:23 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:23.899+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:24.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:24 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:24.727+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:24.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:24 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:24.486+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:25.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:24 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:24.825+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:25.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:24 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:24.898+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:25.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:25 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:25.681+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:25.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:25 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:25.470+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:26.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:25 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:25.839+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:26.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:25 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:25.857+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:26.706 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:26 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:26.432+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:27.130 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:26 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:26.703+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:27.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:26 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:26.859+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:27.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:26 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:26.845+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:27.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:27 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:27.709+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:27.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:27 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:27.464+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:27.901 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:27 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:27.541+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:28.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:27 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:27.898+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:28.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:27 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:27.880+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:28.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:27 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:27.880+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:28.705 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:28 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:28.419+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:28.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:28 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:28.513+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:28.711 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 19:05:28 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-4[139487]: 2023-07-14T19:05:28.291+0000 7f303b2de700 -1 osd.4 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.060570+0000 front 2023-07-14T19:05:03.060596+0000 (oldest deadline 2023-07-14T19:05:27.760734+0000) 2023-07-14T19:05:29.130 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:28 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:28.702+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:29.211 INFO:journalctl@ceph.osd.4.smithi088.stdout:Jul 14 19:05:29 smithi088 bash[156815]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-4 2023-07-14T19:05:29.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:28 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:28.923+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:29.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:29 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:29.110+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:29.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:28 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:28.896+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:29.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:28 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:28.896+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:29.802 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:29 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:29.480+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:29.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:29 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:29.677+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:29.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:29 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:29.439+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:30.184 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:29 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:29.933+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:30.185 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:29 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:29.933+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:30.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:30 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:30.127+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:30.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:29 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:29.882+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:30.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:29 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:29.882+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:30.490 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.4.service' 2023-07-14T19:05:30.570 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:05:30.570 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-07-14T19:05:30.571 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-07-14T19:05:30.571 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.5 2023-07-14T19:05:30.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:30 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:30.435+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:30.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:30.450+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:30.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:30 smithi102 systemd[1]: Stopping Ceph osd.5 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:05:30.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:30.645+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:31.130 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:30 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:30.849+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:31.130 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:30 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:30.849+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:31.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:30 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:30.931+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:31.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:30 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:30.931+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:31.380 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:30.930+0000 7fb3041a7700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:05:31.380 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:30.930+0000 7fb3041a7700 -1 osd.5 93 *** Got signal Terminated *** 2023-07-14T19:05:31.381 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:30.930+0000 7fb3041a7700 -1 osd.5 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:05:31.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:30 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:30.898+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:31.546 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:31 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:31.126+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:31.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:31 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:31.452+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:31.877 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:31.411+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:31.878 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:31.630+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:31.878 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:31.630+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.195344+0000 front 2023-07-14T19:05:05.195142+0000 (oldest deadline 2023-07-14T19:05:31.095184+0000) 2023-07-14T19:05:32.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:31 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:31.874+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:32.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:31 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:31.944+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:32.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:31 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:31.944+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:32.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:32 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:32.077+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:32.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:31 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:31.839+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:32.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:31 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:31.839+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:32.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:32 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:32.410+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:32.875 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:32 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:32.419+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:32.875 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:32 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:32.646+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:32.875 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:32 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:32.646+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.195344+0000 front 2023-07-14T19:05:05.195142+0000 (oldest deadline 2023-07-14T19:05:31.095184+0000) 2023-07-14T19:05:33.119 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:32 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:32.856+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:33.120 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:32 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:32.856+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:33.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:32 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:32.871+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:33.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:32 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:32.927+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:33.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:32 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:32.927+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:33.546 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:33 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:33.116+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:33.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:33.401+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:33.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:33 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:33.644+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:33.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:33 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:33.644+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.195344+0000 front 2023-07-14T19:05:05.195142+0000 (oldest deadline 2023-07-14T19:05:31.095184+0000) 2023-07-14T19:05:33.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:33 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:33.412+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:34.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:33.901+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:34.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:33 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:33.901+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:34.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:34 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:34.140+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:34.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:33 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:33.903+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:34.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:33 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:33.903+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:34.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:33 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:33.918+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:34.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:34 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:34.372+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:34.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:34.452+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:34.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:34.690+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:34.881 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:34.690+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.195344+0000 front 2023-07-14T19:05:05.195142+0000 (oldest deadline 2023-07-14T19:05:31.095184+0000) 2023-07-14T19:05:34.881 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:34.691+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:11.095540+0000 front 2023-07-14T19:05:11.095558+0000 (oldest deadline 2023-07-14T19:05:34.595674+0000) 2023-07-14T19:05:35.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:34 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:34.875+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:35.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:34 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:34.875+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:35.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:35 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:35.116+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:35.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:34 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:34.923+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:35.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:34 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:34.923+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:35.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:34 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:34.962+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:35.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:35 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:35.331+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:35.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:35 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:35.432+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:35.881 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:35 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:35.667+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.693459+0000 front 2023-07-14T19:04:54.693482+0000 (oldest deadline 2023-07-14T19:05:19.393690+0000) 2023-07-14T19:05:35.881 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:35 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:35.667+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.195344+0000 front 2023-07-14T19:05:05.195142+0000 (oldest deadline 2023-07-14T19:05:31.095184+0000) 2023-07-14T19:05:35.882 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:35 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5[140549]: 2023-07-14T19:05:35.667+0000 7fb2fe79a700 -1 osd.5 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:11.095540+0000 front 2023-07-14T19:05:11.095558+0000 (oldest deadline 2023-07-14T19:05:34.595674+0000) 2023-07-14T19:05:36.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:35 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:35.896+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:36.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:35 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:35.896+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:36.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:36 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:36.105+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:36.297 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:36 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:36.105+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.585008+0000 front 2023-07-14T19:05:12.585036+0000 (oldest deadline 2023-07-14T19:05:36.085052+0000) 2023-07-14T19:05:36.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:35 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:35.965+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:36.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:35 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:35.965+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:36.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:35 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:35.965+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:36.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:36.011+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:36.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:36.011+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:36.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:36.318+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:36.880 INFO:journalctl@ceph.osd.5.smithi102.stdout:Jul 14 19:05:36 smithi102 bash[158158]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-5 2023-07-14T19:05:36.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:36.449+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:37.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:36.897+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:37.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:36.897+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:37.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:36 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:36.897+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:37.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:37 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:37.104+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:37.297 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:37 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:37.104+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.585008+0000 front 2023-07-14T19:05:12.585036+0000 (oldest deadline 2023-07-14T19:05:36.085052+0000) 2023-07-14T19:05:37.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:36 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:36.922+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:37.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:36 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:36.922+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:37.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:36 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:36.922+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:37.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:36.966+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:37.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:36 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:36.966+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:37.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:37 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:37.339+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:37.778 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:37 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:37.495+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:37.778 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:37 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:37.495+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:37.784 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.5.service' 2023-07-14T19:05:37.825 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:05:37.826 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-07-14T19:05:37.826 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-07-14T19:05:37.826 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.6 2023-07-14T19:05:38.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:37 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:37.920+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:38.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:37 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:37.920+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:38.187 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:37 smithi078 systemd[1]: Stopping Ceph osd.6 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:05:38.188 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:38 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:38.103+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:38.188 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:38 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:38.103+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.585008+0000 front 2023-07-14T19:05:12.585036+0000 (oldest deadline 2023-07-14T19:05:36.085052+0000) 2023-07-14T19:05:38.188 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:37 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:37.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:38.189 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:37 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:37.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:38.189 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:37 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:37.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:38.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:37 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:37.848+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:38.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:37 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:37.848+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:38.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:37 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:37.848+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:38.546 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:38 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:38.184+0000 7f3c7fa00700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:05:38.547 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:38 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:38.184+0000 7f3c7fa00700 -1 osd.6 93 *** Got signal Terminated *** 2023-07-14T19:05:38.547 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:38 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:38.184+0000 7f3c7fa00700 -1 osd.6 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:05:38.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:38.297+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:38.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:38.297+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:38.875 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:38 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:38.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:38.875 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:38 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:38.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:39.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:38 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:38.872+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:39.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:38 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:38.872+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:39.145 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:38 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:38.891+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:39.145 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:38 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:38.891+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:39.146 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:38 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:38.891+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:39.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:38.870+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:39.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:38.870+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:39.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:38 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:38.870+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:39.546 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:39 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:39.141+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:39.547 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:39 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:39.141+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.585008+0000 front 2023-07-14T19:05:12.585036+0000 (oldest deadline 2023-07-14T19:05:36.085052+0000) 2023-07-14T19:05:39.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:39 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:39.320+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:39.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:39 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:39.320+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:39.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:39 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:39.543+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:39.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:39 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:39.543+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:40.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:39 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:39.893+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:40.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:39 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:39.893+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:40.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:39 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:39.893+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:40.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:40 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:40.121+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:40.297 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:40 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:40.121+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.585008+0000 front 2023-07-14T19:05:12.585036+0000 (oldest deadline 2023-07-14T19:05:36.085052+0000) 2023-07-14T19:05:40.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:39 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:39.875+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:40.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:39 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:39.875+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:40.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:39 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:39.875+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:40.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:39 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:39.888+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:40.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:39 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:39.888+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:40.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:40 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:40.367+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:40.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:40 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:40.367+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:40.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:40 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:40.530+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:40.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:40 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:40.530+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:41.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:40 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:40.885+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:41.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:40 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:40.885+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:41.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:40 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:40.885+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:41.297 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:41 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:41.076+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:41.297 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:41 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:41.076+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.585008+0000 front 2023-07-14T19:05:12.585036+0000 (oldest deadline 2023-07-14T19:05:36.085052+0000) 2023-07-14T19:05:41.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:40 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:40.915+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:41.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:40 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:40.915+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:41.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:40 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:40.915+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:41.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:40 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:40.931+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:41.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:40 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:40.931+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:41.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:41 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:41.406+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:41.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:41 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:41.406+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:41.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:41 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:41.564+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:41.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:41 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:41.564+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:42.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:41 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:41.900+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:42.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:41 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:41.900+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:42.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:41 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:41.900+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:42.296 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:42.045+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:42.297 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:42.046+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.585008+0000 front 2023-07-14T19:05:12.585036+0000 (oldest deadline 2023-07-14T19:05:36.085052+0000) 2023-07-14T19:05:42.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:41 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:41.895+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:42.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:41 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:41.895+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:42.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:41 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:41.895+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:42.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:41 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:41.889+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:42.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:41 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:41.889+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:42.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:42.443+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:42.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:42.443+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:42.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:42.443+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:42.869 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:42.534+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:42.869 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:42.534+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:43.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:42.865+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:43.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:42 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:42.865+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:43.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:42.856+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:43.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:42.856+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:43.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:42.856+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:43.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:42.856+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:43.297 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:43.040+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:03.283450+0000 front 2023-07-14T19:05:03.283428+0000 (oldest deadline 2023-07-14T19:05:28.583485+0000) 2023-07-14T19:05:43.297 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:43.040+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.585008+0000 front 2023-07-14T19:05:12.585036+0000 (oldest deadline 2023-07-14T19:05:36.085052+0000) 2023-07-14T19:05:43.298 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6[151401]: 2023-07-14T19:05:43.040+0000 7f3c7bff7700 -1 osd.6 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.785899+0000 front 2023-07-14T19:05:17.785859+0000 (oldest deadline 2023-07-14T19:05:42.986581+0000) 2023-07-14T19:05:43.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:42.855+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:43.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:42.855+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:43.299 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:42.855+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:43.299 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:42 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:42.855+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:43.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:43.443+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:43.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:43.443+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:43.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:43.443+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:43.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:43 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:43.523+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:43.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:43 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:43.523+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:44.046 INFO:journalctl@ceph.osd.6.smithi078.stdout:Jul 14 19:05:43 smithi078 bash[181014]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-6 2023-07-14T19:05:44.047 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:43.821+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:44.047 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:43.821+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:44.047 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:43.821+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:44.048 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:43 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:43.821+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:44.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:43.885+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:44.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:43.885+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:44.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:43.885+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:44.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:43 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:43.885+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:44.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:43 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:43.885+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:44.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:43 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:43.885+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:44.840 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:44 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:44.464+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:44.840 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:44 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:44.464+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:44.840 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:44 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:44.464+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:44.856 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:44 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:44.503+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:44.856 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:44 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:44.503+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:45.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:44 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:44.852+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:45.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:44 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:44.852+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:45.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:44 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:44.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:45.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:44 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:44.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:45.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:44 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:44.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:45.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:44 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:44.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:45.226 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:44 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:44.854+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:45.227 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:44 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:44.854+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:45.227 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:44 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:44.854+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:45.227 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:44 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:44.854+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:45.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.6.service' 2023-07-14T19:05:45.272 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:05:45.272 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-07-14T19:05:45.273 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-07-14T19:05:45.273 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.7 2023-07-14T19:05:45.626 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:45 smithi088 systemd[1]: Stopping Ceph osd.7 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:05:45.627 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:45.415+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:45.627 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:45.415+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:45.627 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:45.415+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:45.861 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:45 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:45.529+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:45.862 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:45 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:45.529+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:45.862 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:45 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:45.529+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:45.961 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:45.821+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:45.961 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:45.821+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:45.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:45.821+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:45.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:45.821+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:45.962 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:45.624+0000 7f798c902700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:05:45.962 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:45.624+0000 7f798c902700 -1 osd.7 93 *** Got signal Terminated *** 2023-07-14T19:05:45.963 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:45 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:45.624+0000 7f798c902700 -1 osd.7 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:05:46.129 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:45 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:45.858+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:46.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:45 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:45.858+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:46.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:45 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:45.858+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:46.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:45 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:45.872+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:46.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:45 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:45.872+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:46.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:45 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:45.872+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:46.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:45 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:45.872+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:46.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:46 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:46.394+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:46.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:46 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:46.394+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:46.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:46 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:46.394+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:46.859 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:46 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:46.526+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:46.859 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:46 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:46.526+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:46.859 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:46 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:46.526+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:47.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:46 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:46.856+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:47.131 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:46 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:46.856+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:47.131 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:46 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:46.856+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:47.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:46 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:46.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:47.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:46 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:46.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:47.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:46 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:46.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:47.213 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:46 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:46.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:47.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:46 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:46.912+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:47.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:46 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:46.912+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:47.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:46 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:46.912+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:47.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:46 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:46.912+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:47.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:47.373+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:47.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:47.373+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:47.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:47.373+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:47.868 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:47 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:47.572+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:47.869 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:47 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:47.572+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:47.869 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:47 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:47.572+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:48.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:47 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:47.865+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:48.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:47 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:47.865+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:48.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:47 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:47.865+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:48.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:47.843+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:48.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:47.843+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:48.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:47.843+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:48.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:47.843+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:48.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:47.916+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:48.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:47.916+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:48.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:47.916+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:48.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:47 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:47.916+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:48.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:48.409+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:48.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:48.409+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:48.713 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:48.409+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:48.868 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:48 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:48.550+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:48.868 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:48 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:48.550+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:48.868 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:48 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:48.550+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:49.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:48 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:48.865+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:49.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:48 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:48.865+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:49.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:48 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:48.865+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:49.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:48.870+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:49.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:48.870+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:49.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:48.870+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:49.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:48 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:48.870+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:49.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:48 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:48.900+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:49.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:48 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:48.900+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:49.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:48 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:48.900+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:49.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:48 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:48.900+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:49.712 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:49 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:49.442+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:49.713 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:49 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:49.442+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:49.713 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:49 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:49.442+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:49.873 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:49.591+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:49.873 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:49.591+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:49.874 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:49.591+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:49.874 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:49.591+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:50.132 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:49.870+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:50.133 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:49.870+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:50.133 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:49.870+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:50.133 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:49 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:49.870+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:50.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:49 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:49.866+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:50.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:49 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:49.866+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:50.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:49 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:49.866+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:50.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:49 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:49.866+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:50.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:49 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:49.866+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:50.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:49 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:49.934+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:50.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:49 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:49.934+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:50.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:49 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:49.934+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:50.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:49 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:49.934+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:50.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:50 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:50.549+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:50.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:50 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:50.550+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:50.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:50 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:50.550+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:50.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:50 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:50.550+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:50.892 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:50.460+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:01.111876+0000 front 2023-07-14T19:05:01.111857+0000 (oldest deadline 2023-07-14T19:05:27.011963+0000) 2023-07-14T19:05:50.893 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:50.460+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.913200+0000 front 2023-07-14T19:05:12.913117+0000 (oldest deadline 2023-07-14T19:05:37.613131+0000) 2023-07-14T19:05:50.893 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:50.460+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.613670+0000 front 2023-07-14T19:05:17.613656+0000 (oldest deadline 2023-07-14T19:05:41.713565+0000) 2023-07-14T19:05:50.893 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7[144887]: 2023-07-14T19:05:50.460+0000 7f7988ef9700 -1 osd.7 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:26.914965+0000 front 2023-07-14T19:05:26.914992+0000 (oldest deadline 2023-07-14T19:05:50.315804+0000) 2023-07-14T19:05:51.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:50.890+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:51.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:50.890+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:51.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:50.890+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:51.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:50.890+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:51.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:50 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:50.890+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:51.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:50.915+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:51.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:50.915+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:51.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:50.915+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:51.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:50 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:50.915+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:51.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:50 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:50.904+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:51.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:50 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:50.904+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:51.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:50 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:50.904+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:51.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:50 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:50.904+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:51.711 INFO:journalctl@ceph.osd.7.smithi088.stdout:Jul 14 19:05:51 smithi088 bash[157076]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-7 2023-07-14T19:05:51.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:51 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:51.520+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:51.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:51 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:51.520+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:51.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:51 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:51.520+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:51.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:51 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:51.520+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:52.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:51 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:51.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:52.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:51 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:51.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:52.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:51 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:51.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:52.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:51 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:51.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:52.213 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:51 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:51.858+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:52.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:51 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:51.945+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:52.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:51 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:51.945+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:52.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:51 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:51.945+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:52.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:51 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:51.945+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:52.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:51 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:51.945+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:05:52.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:51 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:51.895+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:52.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:51 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:51.895+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:52.382 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:51 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:51.895+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:52.382 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:51 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:51.895+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:52.675 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.7.service' 2023-07-14T19:05:52.716 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:05:52.716 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-07-14T19:05:52.717 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2023-07-14T19:05:52.717 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.8 2023-07-14T19:05:52.880 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:52 smithi102 systemd[1]: Stopping Ceph osd.8 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:05:52.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:52 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:52.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:52.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:52 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:52.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:52.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:52 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:52.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:52.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:52 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:52.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:52.961 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:52 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:52.864+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:52.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:52 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:52.864+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:52.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:52 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:52.864+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:52.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:52 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:52.864+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:52.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:52 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:52.864+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:53.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:52 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:52.935+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:53.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:52 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:52.935+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:53.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:52 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:52.935+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:53.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:52 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:52.935+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:53.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:52 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:52.935+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:05:53.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:52 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:52.899+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:53.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:52 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:52.899+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:53.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:52 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:52.899+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:53.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:52 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:52.899+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:53.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:53.077+0000 7fe6e67b4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:05:53.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:53.077+0000 7fe6e67b4700 -1 osd.8 93 *** Got signal Terminated *** 2023-07-14T19:05:53.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:53.077+0000 7fe6e67b4700 -1 osd.8 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:05:53.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:53.562+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:53.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:53.562+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:53.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:53.562+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:53.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:53.562+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:54.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:53 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:53.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:54.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:53 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:53.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:54.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:53 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:53.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:54.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:53 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:53.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:54.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:53 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:53.837+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:54.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:53.921+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:54.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:53.921+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:54.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:53.921+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:54.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:53.921+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:54.299 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:53 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:53.921+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:05:54.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:53.946+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:54.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:53.946+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:54.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:53.946+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:54.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:53 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:53.946+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:54.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:54 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:54.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:54.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:54 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:54.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:54.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:54 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:54.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:54.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:54 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:54.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:55.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:54 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:54.879+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:55.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:54 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:54.879+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:55.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:54 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:54.879+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:55.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:54 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:54.879+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:55.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:54 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:54.879+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:55.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:54 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:54.965+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:55.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:54 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:54.965+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:55.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:54 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:54.965+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:55.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:54 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:54.965+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:55.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:54 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:54.965+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:05:55.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:54 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:54.915+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:55.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:54 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:54.915+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:55.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:54 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:54.915+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:55.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:54 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:54.915+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:55.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:55 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:55.501+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:55.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:55 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:55.501+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:55.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:55 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:55.501+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:55.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:55 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:55.501+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:56.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:55 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:55.927+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:56.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:55 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:55.927+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:56.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:55 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:55.927+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:56.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:55 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:55.927+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:56.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:55 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:55.927+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:56.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:55 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:55.947+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:56.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:55 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:55.947+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:56.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:55 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:55.947+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:56.299 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:55 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:55.947+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:56.299 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:55 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:55.947+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:05:56.380 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:55 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:55.910+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:56.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:55 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:55.910+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:56.381 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:55 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:55.910+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:56.382 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:55 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:55.910+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:56.866 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:56 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:56.540+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:56.866 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:56 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:56.540+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:56.867 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:56 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:56.540+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:56.867 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:56 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:56.540+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:57.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:56 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:56.863+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:57.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:56 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:56.863+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:57.131 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:56 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:56.863+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:57.132 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:56 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:56.863+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:57.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:56 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:56.902+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:57.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:56 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:56.902+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:57.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:56 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:56.902+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:57.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:56 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:56.903+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:57.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:56 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:56.903+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:05:57.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:56 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:56.903+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:05:57.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:56 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:56.965+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:57.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:56 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:56.965+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:57.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:56 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:56.965+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:57.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:56 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:56.965+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:57.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:56 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:56.965+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:57.846 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:57 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:57.509+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:57.846 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:57 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:57.509+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:57.847 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:57 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:57.509+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:57.848 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:57 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:57.509+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:58.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:57 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:57.843+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.214978+0000 front 2023-07-14T19:05:04.214988+0000 (oldest deadline 2023-07-14T19:05:30.115084+0000) 2023-07-14T19:05:58.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:57 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:57.843+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:10.115445+0000 front 2023-07-14T19:05:10.115479+0000 (oldest deadline 2023-07-14T19:05:35.316164+0000) 2023-07-14T19:05:58.130 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:57 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:57.843+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:21.018510+0000 front 2023-07-14T19:05:21.018361+0000 (oldest deadline 2023-07-14T19:05:45.118446+0000) 2023-07-14T19:05:58.131 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:57 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8[146084]: 2023-07-14T19:05:57.843+0000 7fe6e15a8700 -1 osd.8 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.119222+0000 front 2023-07-14T19:05:25.119246+0000 (oldest deadline 2023-07-14T19:05:49.819062+0000) 2023-07-14T19:05:58.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:57 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:57.918+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:58.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:57 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:57.918+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:58.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:57 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:57.918+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:58.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:57 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:57.918+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:58.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:57 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:57.918+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:58.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:57 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:57.870+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:58.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:57 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:57.870+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:58.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:57 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:57.870+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:58.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:57 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:57.870+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:58.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:57 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:57.870+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:05:58.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:57 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:57.870+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:05:58.803 INFO:journalctl@ceph.osd.8.smithi102.stdout:Jul 14 19:05:58 smithi102 bash[158420]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-8 2023-07-14T19:05:58.804 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:58 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:58.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:58.804 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:58 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:58.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:58.804 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:58 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:58.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:58.804 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:58 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:58.522+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:59.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:58 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:58.952+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:05:59.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:58 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:58.952+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:05:59.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:58 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:58.952+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:05:59.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:58 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:58.952+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:05:59.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:05:58 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:05:58.952+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:05:59.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:58 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:58.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:05:59.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:58 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:58.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:05:59.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:58 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:58.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:05:59.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:58 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:58.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:05:59.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:58 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:58.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:05:59.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:58 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:58.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:05:59.630 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:59 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:59.490+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:05:59.631 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:59 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:59.490+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:05:59.631 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:59 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:59.490+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:05:59.631 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:59 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:59.490+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:05:59.631 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:05:59 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:05:59.490+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:00.064 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.8.service' 2023-07-14T19:06:00.103 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:00.104 INFO:tasks.cephadm.osd.8:Stopped osd.8 2023-07-14T19:06:00.104 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2023-07-14T19:06:00.105 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.9 2023-07-14T19:06:00.135 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:59 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:59.848+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:06:00.135 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:59 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:59.848+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:06:00.135 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:59 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:59.848+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:06:00.135 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:59 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:59.848+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:06:00.136 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:59 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:59.848+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:06:00.136 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:05:59 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:05:59.848+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:06:00.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.001+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:00.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.001+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:00.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.001+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:00.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.001+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:00.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.001+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:00.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.001+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:00.467 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 systemd[1]: Stopping Ceph osd.9 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:00.796 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.464+0000 7f28a0f7c700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:00.797 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.465+0000 7f28a0f7c700 -1 osd.9 93 *** Got signal Terminated *** 2023-07-14T19:06:00.797 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.465+0000 7f28a0f7c700 -1 osd.9 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:06:00.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:00 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:00.532+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:00.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:00 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:00.532+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:00.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:00 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:00.532+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:00.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:00 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:00.532+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:00.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:00 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:00.532+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:01.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:06:01.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:06:01.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:06:01.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:06:01.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:06:01.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:00 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:00.873+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:06:01.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.989+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:01.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.989+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:01.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.989+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:01.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.989+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:01.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.989+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:01.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:00 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:00.989+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:01.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:01 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:01.565+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:01.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:01 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:01.565+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:01.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:01 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:01.565+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:01.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:01 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:01.565+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:01.882 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:01 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:01.565+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:02.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:01.852+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:06:02.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:01.852+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:06:02.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:01.852+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:06:02.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:01.852+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:06:02.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:01.852+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:06:02.299 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:01 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:01.852+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:06:02.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:01 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:01.981+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:02.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:01 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:01.981+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:02.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:01 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:01.981+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:02.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:01 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:01.981+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:02.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:01 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:01.981+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:02.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:01 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:01.981+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:02.879 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:02 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:02.561+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:02.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:02 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:02.561+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:02.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:02 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:02.561+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:02.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:02 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:02.561+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:02.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:02 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:02.561+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:03.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:02.833+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:06:03.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:02.833+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:06:03.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:02.833+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:06:03.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:02.833+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:06:03.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:02.833+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:06:03.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:02 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:02.833+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:06:03.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:03 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:03.015+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:03.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:03 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:03.015+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:03.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:03 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:03.015+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:03.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:03 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:03.015+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:03.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:03 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:03.015+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:03.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:03 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:03.015+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:03.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:03 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:03.572+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:03.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:03 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:03.572+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:03.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:03 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:03.572+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:03.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:03 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:03.572+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:03.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:03 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:03.572+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:04.296 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:03 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:03.840+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:06:04.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:03 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:03.840+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:06:04.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:03 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:03.840+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:06:04.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:03 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:03.840+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:06:04.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:03 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:03.840+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:06:04.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:03 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:03.840+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:06:04.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:04 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:04.032+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:04.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:04 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:04.032+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:04.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:04 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:04.032+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:04.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:04 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:04.032+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:04.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:04 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:04.032+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:04.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:04 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:04.032+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:04.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:04 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:04.577+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:04.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:04 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:04.577+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:04.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:04 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:04.577+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:04.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:04 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:04.577+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:04.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:04 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:04.577+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:05.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:04.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:54.672076+0000 front 2023-07-14T19:04:54.672106+0000 (oldest deadline 2023-07-14T19:05:19.972159+0000) 2023-07-14T19:06:05.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:04.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:04.073420+0000 front 2023-07-14T19:05:04.073346+0000 (oldest deadline 2023-07-14T19:05:26.874108+0000) 2023-07-14T19:06:05.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:04.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.075676+0000 front 2023-07-14T19:05:12.075585+0000 (oldest deadline 2023-07-14T19:05:34.975466+0000) 2023-07-14T19:06:05.297 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:04.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.576839+0000 front 2023-07-14T19:05:19.576879+0000 (oldest deadline 2023-07-14T19:05:42.476864+0000) 2023-07-14T19:06:05.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:04.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:27.177992+0000 front 2023-07-14T19:05:27.178195+0000 (oldest deadline 2023-07-14T19:05:51.277974+0000) 2023-07-14T19:06:05.298 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:04 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9[156881]: 2023-07-14T19:06:04.885+0000 7f289b56f700 -1 osd.9 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:31.278611+0000 front 2023-07-14T19:05:31.278489+0000 (oldest deadline 2023-07-14T19:05:56.578363+0000) 2023-07-14T19:06:05.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:05 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:05.055+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:05.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:05 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:05.055+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:05.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:05 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:05.055+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:05.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:05 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:05.055+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:05.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:05 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:05.055+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:05.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:05 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:05.055+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:05.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:05 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:05.055+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:40.554134+0000 front 2023-07-14T19:05:40.554150+0000 (oldest deadline 2023-07-14T19:06:04.053948+0000) 2023-07-14T19:06:06.130 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:05 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:05.627+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:06.130 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:05 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:05.627+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:06.130 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:05 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:05.627+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:06.131 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:05 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:05.627+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:06.131 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:05 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:05.627+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:06.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:06 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:06.029+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:06.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:06 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:06.029+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:06.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:06 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:06.030+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:06.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:06 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:06.030+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:06.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:06 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:06.030+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:06.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:06 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:06.030+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:06.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:06 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:06.030+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:40.554134+0000 front 2023-07-14T19:05:40.554150+0000 (oldest deadline 2023-07-14T19:06:04.053948+0000) 2023-07-14T19:06:06.546 INFO:journalctl@ceph.osd.9.smithi078.stdout:Jul 14 19:06:06 smithi078 bash[181282]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-9 2023-07-14T19:06:06.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:06 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:06.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:06.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:06 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:06.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:06.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:06 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:06.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:06.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:06 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:06.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:06.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:06 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:06.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:06.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:06 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:06.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:07.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.074+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:07.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.074+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:07.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.074+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:07.464 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.074+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:07.464 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.074+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:07.465 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.074+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:07.465 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.074+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:40.554134+0000 front 2023-07-14T19:05:40.554150+0000 (oldest deadline 2023-07-14T19:06:04.053948+0000) 2023-07-14T19:06:07.502 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.9.service' 2023-07-14T19:06:07.585 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:07.586 INFO:tasks.cephadm.osd.9:Stopped osd.9 2023-07-14T19:06:07.586 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2023-07-14T19:06:07.587 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.10 2023-07-14T19:06:07.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:07 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:07.566+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:07.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:07 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:07.566+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:07.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:07 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:07.566+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:07.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:07 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:07.566+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:07.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:07 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:07.566+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:07.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:07 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:07.566+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:07.956 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 systemd[1]: Stopping Ceph osd.10 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:07.956 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.953+0000 7f487e4d9700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:08.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.953+0000 7f487e4d9700 -1 osd.10 93 *** Got signal Terminated *** 2023-07-14T19:06:08.211 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:07 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:07.953+0000 7f487e4d9700 -1 osd.10 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:06:08.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:08 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:08.081+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:08.212 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:08 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:08.081+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:08.213 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:08 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:08.081+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:08.213 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:08 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:08.081+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:08.213 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:08 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:08.081+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:08.214 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:08 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:08.081+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:08.214 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:08 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:08.081+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:40.554134+0000 front 2023-07-14T19:05:40.554150+0000 (oldest deadline 2023-07-14T19:06:04.053948+0000) 2023-07-14T19:06:08.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:08.589+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:08.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:08.589+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:08.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:08.589+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:08.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:08.589+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:08.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:08.589+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:08.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:08.589+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:09.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:09 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:09.116+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:09.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:09 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:09.116+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:09.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:09 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:09.116+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:09.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:09 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:09.116+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:09.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:09 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:09.116+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:09.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:09 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:09.116+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:09.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:09 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:09.116+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:40.554134+0000 front 2023-07-14T19:05:40.554150+0000 (oldest deadline 2023-07-14T19:06:04.053948+0000) 2023-07-14T19:06:09.879 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:09 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:09.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:09.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:09 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:09.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:09.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:09 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:09.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:09.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:09 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:09.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:09.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:09 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:09.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:09.882 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:09 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:09.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:10.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:10 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:10.144+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:10.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:10 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:10.144+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:10.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:10 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:10.144+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:10.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:10 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:10.144+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:10.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:10 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:10.144+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:10.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:10 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:10.144+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:10.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:10 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:10.144+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:40.554134+0000 front 2023-07-14T19:05:40.554150+0000 (oldest deadline 2023-07-14T19:06:04.053948+0000) 2023-07-14T19:06:10.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:10 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:10.580+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:10.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:10 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:10.580+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:10.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:10 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:10.580+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:10.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:10 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:10.580+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:10.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:10 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:10.580+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:10.882 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:10 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:10.580+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:11.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:11 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:11.178+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:11.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:11 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:11.178+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:11.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:11 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:11.178+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:11.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:11 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:11.178+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:11.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:11 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:11.178+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:11.464 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:11 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:11.178+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:11.464 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:11 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:11.178+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:40.554134+0000 front 2023-07-14T19:05:40.554150+0000 (oldest deadline 2023-07-14T19:06:04.053948+0000) 2023-07-14T19:06:11.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:11.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:11.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:11.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:11.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:11.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:11.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:11.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:11.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:11.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:11.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:11.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:11.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:11.541+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:12.461 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:12 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:12.163+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:56.445936+0000 front 2023-07-14T19:04:56.445823+0000 (oldest deadline 2023-07-14T19:05:22.345806+0000) 2023-07-14T19:06:12.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:12 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:12.163+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6804 osd.1 since back 2023-07-14T19:05:05.747375+0000 front 2023-07-14T19:05:05.747345+0000 (oldest deadline 2023-07-14T19:05:29.847480+0000) 2023-07-14T19:06:12.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:12 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:12.163+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:09.848056+0000 front 2023-07-14T19:05:09.847976+0000 (oldest deadline 2023-07-14T19:05:36.248333+0000) 2023-07-14T19:06:12.462 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:12 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:12.163+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:17.949588+0000 front 2023-07-14T19:05:17.949479+0000 (oldest deadline 2023-07-14T19:05:42.649394+0000) 2023-07-14T19:06:12.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:12 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:12.163+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.550605+0000 front 2023-07-14T19:05:25.550575+0000 (oldest deadline 2023-07-14T19:05:49.050793+0000) 2023-07-14T19:06:12.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:12 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:12.163+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:34.852736+0000 front 2023-07-14T19:05:34.852641+0000 (oldest deadline 2023-07-14T19:05:59.552495+0000) 2023-07-14T19:06:12.463 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:12 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10[150426]: 2023-07-14T19:06:12.163+0000 7f4878acc700 -1 osd.10 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:40.554134+0000 front 2023-07-14T19:05:40.554150+0000 (oldest deadline 2023-07-14T19:06:04.053948+0000) 2023-07-14T19:06:12.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:12 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:12.579+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:12.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:12 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:12.579+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:12.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:12 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:12.579+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:12.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:12 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:12.579+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:12.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:12 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:12.579+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:12.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:12 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:12.579+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:12.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:12 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:12.579+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:13.879 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:13 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:13.552+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:13.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:13 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:13.552+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:13.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:13 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:13.552+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:13.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:13 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:13.552+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:13.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:13 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:13.552+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:13.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:13 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:13.552+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:13.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:13 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:13.552+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:13.962 INFO:journalctl@ceph.osd.10.smithi088.stdout:Jul 14 19:06:13 smithi088 bash[157351]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-10 2023-07-14T19:06:14.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:14 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:14.521+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:14.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:14 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:14.521+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:14.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:14 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:14.521+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:14.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:14 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:14.521+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:14.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:14 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:14.521+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:14.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:14 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:14.521+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:14.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:14 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:14.521+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:15.079 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.10.service' 2023-07-14T19:06:15.120 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:15.120 INFO:tasks.cephadm.osd.10:Stopped osd.10 2023-07-14T19:06:15.121 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2023-07-14T19:06:15.121 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.11 2023-07-14T19:06:15.468 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 systemd[1]: Stopping Ceph osd.11 for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:15.469 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.465+0000 7f41c2665700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:15.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.465+0000 7f41c2665700 -1 osd.11 93 *** Got signal Terminated *** 2023-07-14T19:06:15.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.465+0000 7f41c2665700 -1 osd.11 93 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-07-14T19:06:15.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:15.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:15.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:15.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:15.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:15.882 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:15.882 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:15 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:15.547+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:16.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:16 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:16.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:16.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:16 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:16.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:16.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:16 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:16.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:16.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:16 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:16.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:16.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:16 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:16.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:16.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:16 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:16.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:16.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:16 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:16.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:17.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:17 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:17.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:17.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:17 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:17.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:17.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:17 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:17.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:17.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:17 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:17.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:17.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:17 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:17.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:17.882 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:17 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:17.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:17.882 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:17 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:17.513+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:18.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:18 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:18.555+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:18.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:18 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:18.555+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:18.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:18 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:18.555+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:18.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:18 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:18.555+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:18.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:18 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:18.555+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:18.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:18 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:18.555+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:18.882 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:18 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:18.555+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:19.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:19 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:19.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6806 osd.0 since back 2023-07-14T19:04:53.892756+0000 front 2023-07-14T19:04:53.892700+0000 (oldest deadline 2023-07-14T19:05:19.792662+0000) 2023-07-14T19:06:19.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:19 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:19.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6804 osd.2 since back 2023-07-14T19:05:12.496194+0000 front 2023-07-14T19:05:12.496221+0000 (oldest deadline 2023-07-14T19:05:37.196550+0000) 2023-07-14T19:06:19.880 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:19 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:19.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6814 osd.3 since back 2023-07-14T19:05:19.398241+0000 front 2023-07-14T19:05:19.398183+0000 (oldest deadline 2023-07-14T19:05:45.298170+0000) 2023-07-14T19:06:19.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:19 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:19.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6812 osd.4 since back 2023-07-14T19:05:25.298741+0000 front 2023-07-14T19:05:25.298805+0000 (oldest deadline 2023-07-14T19:05:49.398720+0000) 2023-07-14T19:06:19.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:19 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:19.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.102:6812 osd.5 since back 2023-07-14T19:05:35.299762+0000 front 2023-07-14T19:05:35.299797+0000 (oldest deadline 2023-07-14T19:05:59.399901+0000) 2023-07-14T19:06:19.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:19 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:19.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.78:6822 osd.6 since back 2023-07-14T19:05:39.400356+0000 front 2023-07-14T19:05:39.400499+0000 (oldest deadline 2023-07-14T19:06:06.400764+0000) 2023-07-14T19:06:19.881 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:19 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11[151541]: 2023-07-14T19:06:19.604+0000 7f41bac54700 -1 osd.11 93 heartbeat_check: no reply from 172.21.15.88:6820 osd.7 since back 2023-07-14T19:05:50.501794+0000 front 2023-07-14T19:05:50.501825+0000 (oldest deadline 2023-07-14T19:06:11.001796+0000) 2023-07-14T19:06:21.130 INFO:journalctl@ceph.osd.11.smithi102.stdout:Jul 14 19:06:21 smithi102 bash[158686]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-osd-11 2023-07-14T19:06:22.323 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@osd.11.service' 2023-07-14T19:06:22.363 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:22.364 INFO:tasks.cephadm.osd.11:Stopped osd.11 2023-07-14T19:06:22.364 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2023-07-14T19:06:22.365 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.a 2023-07-14T19:06:22.724 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:06:22 smithi078 systemd[1]: Stopping Ceph mds.a for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:23.046 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:06:22 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-a[162165]: 2023-07-14T19:06:22.720+0000 7faaf3ebc700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:23.047 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:06:22 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-a[162165]: 2023-07-14T19:06:22.720+0000 7faaf3ebc700 -1 mds.a *** got signal Terminated *** 2023-07-14T19:06:27.235 INFO:journalctl@ceph.mds.a.smithi078.stdout:Jul 14 19:06:26 smithi078 bash[181553]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-a 2023-07-14T19:06:27.241 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.a.service' 2023-07-14T19:06:27.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:27.278 INFO:tasks.cephadm.mds.a:Stopped mds.a 2023-07-14T19:06:27.279 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2023-07-14T19:06:27.279 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.d 2023-07-14T19:06:27.546 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:06:27 smithi078 systemd[1]: Stopping Ceph mds.d for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:28.046 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:06:27 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-d[161202]: 2023-07-14T19:06:27.643+0000 7f4139916700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:28.047 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:06:27 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-d[161202]: 2023-07-14T19:06:27.643+0000 7f4139916700 -1 mds.d *** got signal Terminated *** 2023-07-14T19:06:31.796 INFO:journalctl@ceph.mds.d.smithi078.stdout:Jul 14 19:06:31 smithi078 bash[181705]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-d 2023-07-14T19:06:31.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.d.service' 2023-07-14T19:06:31.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:31.953 INFO:tasks.cephadm.mds.d:Stopped mds.d 2023-07-14T19:06:31.953 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2023-07-14T19:06:31.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.g 2023-07-14T19:06:32.296 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:06:32 smithi078 systemd[1]: Stopping Ceph mds.g for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:32.796 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:06:32 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-g[161780]: 2023-07-14T19:06:32.321+0000 7f5c2453a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:32.797 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:06:32 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-g[161780]: 2023-07-14T19:06:32.321+0000 7f5c2453a700 -1 mds.g *** got signal Terminated *** 2023-07-14T19:06:36.796 INFO:journalctl@ceph.mds.g.smithi078.stdout:Jul 14 19:06:36 smithi078 bash[181845]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-g 2023-07-14T19:06:36.875 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.g.service' 2023-07-14T19:06:36.909 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:36.910 INFO:tasks.cephadm.mds.g:Stopped mds.g 2023-07-14T19:06:36.910 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2023-07-14T19:06:36.911 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.j 2023-07-14T19:06:37.280 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:06:36 smithi078 systemd[1]: Stopping Ceph mds.j for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:37.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:06:37 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-j[160816]: 2023-07-14T19:06:37.277+0000 7f49f99ff700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:37.546 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:06:37 smithi078 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-j[160816]: 2023-07-14T19:06:37.277+0000 7f49f99ff700 -1 mds.j *** got signal Terminated *** 2023-07-14T19:06:41.738 INFO:journalctl@ceph.mds.j.smithi078.stdout:Jul 14 19:06:41 smithi078 bash[181990]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-j 2023-07-14T19:06:41.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.j.service' 2023-07-14T19:06:41.863 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:41.863 INFO:tasks.cephadm.mds.j:Stopped mds.j 2023-07-14T19:06:41.864 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2023-07-14T19:06:41.864 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.b 2023-07-14T19:06:42.211 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:06:41 smithi088 systemd[1]: Stopping Ceph mds.b for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:42.711 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:06:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b[154122]: 2023-07-14T19:06:42.229+0000 7fec44e33700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:42.711 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:06:42 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b[154122]: 2023-07-14T19:06:42.229+0000 7fec44e33700 -1 mds.b *** got signal Terminated *** 2023-07-14T19:06:46.711 INFO:journalctl@ceph.mds.b.smithi088.stdout:Jul 14 19:06:46 smithi088 bash[157626]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-b 2023-07-14T19:06:46.758 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.b.service' 2023-07-14T19:06:46.793 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:46.794 INFO:tasks.cephadm.mds.b:Stopped mds.b 2023-07-14T19:06:46.794 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2023-07-14T19:06:46.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.e 2023-07-14T19:06:47.180 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:06:46 smithi088 systemd[1]: Stopping Ceph mds.e for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:47.460 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:06:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-e[153444]: 2023-07-14T19:06:47.177+0000 7fc949d2a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:47.461 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:06:47 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-e[153444]: 2023-07-14T19:06:47.177+0000 7fc949d2a700 -1 mds.e *** got signal Terminated *** 2023-07-14T19:06:51.347 INFO:journalctl@ceph.mds.e.smithi088.stdout:Jul 14 19:06:51 smithi088 bash[157770]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-e 2023-07-14T19:06:51.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.e.service' 2023-07-14T19:06:51.570 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:51.571 INFO:tasks.cephadm.mds.e:Stopped mds.e 2023-07-14T19:06:51.572 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2023-07-14T19:06:51.572 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.h 2023-07-14T19:06:51.953 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:06:51 smithi088 systemd[1]: Stopping Ceph mds.h for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:52.211 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:06:51 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-h[153782]: 2023-07-14T19:06:51.950+0000 7f70e909a700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:52.211 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:06:51 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-h[153782]: 2023-07-14T19:06:51.950+0000 7f70e909a700 -1 mds.h *** got signal Terminated *** 2023-07-14T19:06:55.600 INFO:journalctl@ceph.mds.h.smithi088.stdout:Jul 14 19:06:55 smithi088 bash[157908]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-h 2023-07-14T19:06:55.605 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.h.service' 2023-07-14T19:06:55.638 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:55.639 INFO:tasks.cephadm.mds.h:Stopped mds.h 2023-07-14T19:06:55.640 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2023-07-14T19:06:55.641 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.k 2023-07-14T19:06:55.961 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:06:55 smithi088 systemd[1]: Stopping Ceph mds.k for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:06:56.461 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:06:56 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-k[153101]: 2023-07-14T19:06:56.033+0000 7fc94fe84700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:06:56.462 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:06:56 smithi088 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-k[153101]: 2023-07-14T19:06:56.033+0000 7fc94fe84700 -1 mds.k *** got signal Terminated *** 2023-07-14T19:06:59.350 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:06:59 smithi088 bash[158050]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-k 2023-07-14T19:06:59.580 DEBUG:teuthology.orchestra.run.smithi088:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.k.service' 2023-07-14T19:06:59.605 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:06:59 smithi088 systemd[1]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.k.service: Succeeded. 2023-07-14T19:06:59.605 INFO:journalctl@ceph.mds.k.smithi088.stdout:Jul 14 19:06:59 smithi088 systemd[1]: Stopped Ceph mds.k for e2ca1a0e-2277-11ee-9b33-001a4aab830c. 2023-07-14T19:06:59.615 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:06:59.615 INFO:tasks.cephadm.mds.k:Stopped mds.k 2023-07-14T19:06:59.616 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2023-07-14T19:06:59.616 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.c 2023-07-14T19:06:59.981 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:06:59 smithi102 systemd[1]: Stopping Ceph mds.c for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:07:00.379 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:06:59 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-c[155129]: 2023-07-14T19:06:59.977+0000 7ff8a11d4700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:07:00.380 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:06:59 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-c[155129]: 2023-07-14T19:06:59.977+0000 7ff8a11d4700 -1 mds.c *** got signal Terminated *** 2023-07-14T19:07:03.379 INFO:journalctl@ceph.mds.c.smithi102.stdout:Jul 14 19:07:03 smithi102 bash[158959]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-c 2023-07-14T19:07:03.431 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.c.service' 2023-07-14T19:07:03.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:07:03.469 INFO:tasks.cephadm.mds.c:Stopped mds.c 2023-07-14T19:07:03.470 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2023-07-14T19:07:03.470 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.f 2023-07-14T19:07:03.846 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:07:03 smithi102 systemd[1]: Stopping Ceph mds.f for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:07:04.129 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:07:03 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-f[154788]: 2023-07-14T19:07:03.843+0000 7f52f7e68700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:07:04.130 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:07:03 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-f[154788]: 2023-07-14T19:07:03.843+0000 7f52f7e68700 -1 mds.f *** got signal Terminated *** 2023-07-14T19:07:07.590 INFO:journalctl@ceph.mds.f.smithi102.stdout:Jul 14 19:07:07 smithi102 bash[159096]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-f 2023-07-14T19:07:07.595 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.f.service' 2023-07-14T19:07:07.631 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:07:07.632 INFO:tasks.cephadm.mds.f:Stopped mds.f 2023-07-14T19:07:07.632 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2023-07-14T19:07:07.632 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.i 2023-07-14T19:07:07.879 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:07:07 smithi102 systemd[1]: Stopping Ceph mds.i for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:07:08.380 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:07:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-i[155470]: 2023-07-14T19:07:08.000+0000 7fe802855700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:07:08.380 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:07:08 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-i[155470]: 2023-07-14T19:07:08.000+0000 7fe802855700 -1 mds.i *** got signal Terminated *** 2023-07-14T19:07:11.379 INFO:journalctl@ceph.mds.i.smithi102.stdout:Jul 14 19:07:11 smithi102 bash[159238]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-i 2023-07-14T19:07:11.452 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.i.service' 2023-07-14T19:07:11.485 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:07:11.486 INFO:tasks.cephadm.mds.i:Stopped mds.i 2023-07-14T19:07:11.486 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2023-07-14T19:07:11.487 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.l 2023-07-14T19:07:11.870 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:07:11 smithi102 systemd[1]: Stopping Ceph mds.l for e2ca1a0e-2277-11ee-9b33-001a4aab830c... 2023-07-14T19:07:12.129 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:07:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-l[154448]: 2023-07-14T19:07:11.867+0000 7f6c40e9d700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-07-14T19:07:12.130 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:07:11 smithi102 ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-l[154448]: 2023-07-14T19:07:11.867+0000 7f6c40e9d700 -1 mds.l *** got signal Terminated *** 2023-07-14T19:07:16.380 INFO:journalctl@ceph.mds.l.smithi102.stdout:Jul 14 19:07:16 smithi102 bash[159377]: ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c-mds-l 2023-07-14T19:07:16.399 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-e2ca1a0e-2277-11ee-9b33-001a4aab830c@mds.l.service' 2023-07-14T19:07:16.439 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-07-14T19:07:16.440 INFO:tasks.cephadm.mds.l:Stopped mds.l 2023-07-14T19:07:16.440 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:07:16.748 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T19:07:18.296 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:07:18.594 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T19:07:20.154 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c --force --keep-logs 2023-07-14T19:07:20.457 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T19:07:22.012 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:07:22.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:07:22.086 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-07-14T19:07:22.115 INFO:tasks.cephadm:Archiving crash dumps... 2023-07-14T19:07:22.117 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi078/crash 2023-07-14T19:07:22.118 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:07:22.157 INFO:teuthology.orchestra.run.smithi078.stderr:tar: /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:07:22.158 INFO:teuthology.orchestra.run.smithi078.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:07:22.159 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi088/crash 2023-07-14T19:07:22.160 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:07:22.208 INFO:teuthology.orchestra.run.smithi088.stderr:tar: /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:07:22.209 INFO:teuthology.orchestra.run.smithi088.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:07:22.210 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi102/crash 2023-07-14T19:07:22.211 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash -- . 2023-07-14T19:07:22.244 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/crash: Cannot open: No such file or directory 2023-07-14T19:07:22.245 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2023-07-14T19:07:22.246 INFO:tasks.cephadm:Checking cluster log for badness... 2023-07-14T19:07:22.246 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-07-14T19:07:22.282 INFO:teuthology.orchestra.run.smithi078.stderr:grep: /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph.log: No such file or directory 2023-07-14T19:07:22.284 INFO:tasks.cephadm:Compressing logs... 2023-07-14T19:07:22.284 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:07:22.327 DEBUG:teuthology.orchestra.run.smithi088:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:07:22.330 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-07-14T19:07:22.358 INFO:teuthology.orchestra.run.smithi078.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:07:22.359 INFO:teuthology.orchestra.run.smithi078.stderr:: No such file or directory 2023-07-14T19:07:22.359 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.0.log 2023-07-14T19:07:22.360 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.g.log 2023-07-14T19:07:22.360 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.a.log 2023-07-14T19:07:22.361 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.6.log 2023-07-14T19:07:22.361 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.3.log 2023-07-14T19:07:22.365 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.d.log 2023-07-14T19:07:22.366 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.3.log: /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.d.log: 94.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.g.log.gz 2023-07-14T19:07:22.366 INFO:teuthology.orchestra.run.smithi078.stderr: 94.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.a.log.gz 2023-07-14T19:07:22.366 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.x.log 2023-07-14T19:07:22.366 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.9.log 2023-07-14T19:07:22.367 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.x.log: 94.6% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.d.log.gz 2023-07-14T19:07:22.367 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.9.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:07:22.367 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:07:22.367 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.j.log 2023-07-14T19:07:22.369 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.j.log: 94.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.j.log.gz 2023-07-14T19:07:22.369 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.a.log 2023-07-14T19:07:22.369 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2023-07-14T19:07:22.370 INFO:teuthology.orchestra.run.smithi088.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-07-14T19:07:22.370 INFO:teuthology.orchestra.run.smithi088.stderr:: No such file or directory 2023-07-14T19:07:22.370 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.10.log 2023-07-14T19:07:22.370 INFO:teuthology.orchestra.run.smithi088.stderr:gzip /var/log/ceph/cephadm.log: -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.h.log 2023-07-14T19:07:22.371 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.e.log 2023-07-14T19:07:22.371 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.4.log 2023-07-14T19:07:22.371 INFO:teuthology.orchestra.run.smithi088.stderr: 91.6% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:07:22.371 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.1.log 2023-07-14T19:07:22.371 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.k.log 2023-07-14T19:07:22.372 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:07:22.372 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.k.log: /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.7.log 2023-07-14T19:07:22.372 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log : No such file or directory 2023-07-14T19:07:22.372 INFO:teuthology.orchestra.run.smithi102.stderr: 2023-07-14T19:07:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.5.log 2023-07-14T19:07:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.2.log 2023-07-14T19:07:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.11.log 2023-07-14T19:07:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.z.log 2023-07-14T19:07:22.373 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.8.log 2023-07-14T19:07:22.374 INFO:teuthology.orchestra.run.smithi102.stderr: 92.0% -- replaced with /var/log/ceph/cephadm.log.gz 2023-07-14T19:07:22.374 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.f.log 2023-07-14T19:07:22.374 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.c.log 2023-07-14T19:07:22.374 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.f.log: /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.c.log: 91.7% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.z.log.gz 2023-07-14T19:07:22.374 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.i.log 2023-07-14T19:07:22.375 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.c.log 2023-07-14T19:07:22.375 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log 2023-07-14T19:07:22.376 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.7.log: 94.6% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.k.log.gz 2023-07-14T19:07:22.376 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.b.log 2023-07-14T19:07:22.376 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.b.log 2023-07-14T19:07:22.377 INFO:teuthology.orchestra.run.smithi088.stderr: 91.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:07:22.377 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.a.log: /var/log/ceph/ceph-client.admin.log: 91.6% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:07:22.378 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.log 2023-07-14T19:07:22.378 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.0.log: 94.0% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2023-07-14T19:07:22.378 INFO:teuthology.orchestra.run.smithi078.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.0.log.gz 2023-07-14T19:07:22.379 INFO:teuthology.orchestra.run.smithi102.stderr: 92.7% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.f.log.gz 2023-07-14T19:07:22.379 INFO:teuthology.orchestra.run.smithi088.stderr: 92.8% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.e.log.gz 2023-07-14T19:07:22.380 INFO:teuthology.orchestra.run.smithi088.stderr: 91.3% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.h.log.gz 2023-07-14T19:07:22.381 INFO:teuthology.orchestra.run.smithi088.stderr:gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.y.log 2023-07-14T19:07:22.385 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.l.log 2023-07-14T19:07:22.385 INFO:teuthology.orchestra.run.smithi102.stderr: 92.6% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.c.log.gz 2023-07-14T19:07:22.386 INFO:teuthology.orchestra.run.smithi102.stderr: 91.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.i.log.gz 2023-07-14T19:07:22.386 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.l.log: 91.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-volume.log.gz 2023-07-14T19:07:22.386 INFO:teuthology.orchestra.run.smithi088.stderr:/var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.b.log: /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.y.log: 91.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.y.log.gz 2023-07-14T19:07:22.387 INFO:teuthology.orchestra.run.smithi102.stderr: 94.9% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.l.log.gz 2023-07-14T19:07:22.471 INFO:teuthology.orchestra.run.smithi088.stderr: 92.7% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mds.b.log.gz 2023-07-14T19:07:22.512 INFO:teuthology.orchestra.run.smithi102.stderr: 92.0% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.c.log.gz 2023-07-14T19:07:22.513 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mgr.x.log.gz 2023-07-14T19:07:22.582 INFO:teuthology.orchestra.run.smithi078.stderr: 92.6% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.9.log.gz 2023-07-14T19:07:22.588 INFO:teuthology.orchestra.run.smithi088.stderr: 92.0% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.b.log.gz 2023-07-14T19:07:22.595 INFO:teuthology.orchestra.run.smithi102.stderr: 92.4% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.8.log.gz 2023-07-14T19:07:22.614 INFO:teuthology.orchestra.run.smithi102.stderr: 92.9% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.11.log.gz 2023-07-14T19:07:22.625 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.4.log.gz 2023-07-14T19:07:22.628 INFO:teuthology.orchestra.run.smithi088.stderr: 92.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.7.log.gz 2023-07-14T19:07:22.654 INFO:teuthology.orchestra.run.smithi088.stderr: 92.4% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.10.log.gz 2023-07-14T19:07:22.671 INFO:teuthology.orchestra.run.smithi102.stderr: 92.3% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.2.log.gz 2023-07-14T19:07:22.679 INFO:teuthology.orchestra.run.smithi078.stderr: 92.4% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.3.log.gz 2023-07-14T19:07:22.684 INFO:teuthology.orchestra.run.smithi078.stderr: 92.4% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.6.log.gz 2023-07-14T19:07:22.715 INFO:teuthology.orchestra.run.smithi078.stderr: 92.5% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.0.log.gz 2023-07-14T19:07:22.719 INFO:teuthology.orchestra.run.smithi102.stderr: 92.4% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.5.log.gz 2023-07-14T19:07:22.721 INFO:teuthology.orchestra.run.smithi102.stderr: 2023-07-14T19:07:22.721 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m0.375s 2023-07-14T19:07:22.722 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m1.340s 2023-07-14T19:07:22.722 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m0.073s 2023-07-14T19:07:22.769 INFO:teuthology.orchestra.run.smithi088.stderr: 92.2% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-osd.1.log.gz 2023-07-14T19:07:22.770 INFO:teuthology.orchestra.run.smithi088.stderr: 2023-07-14T19:07:22.770 INFO:teuthology.orchestra.run.smithi088.stderr:real 0m0.414s 2023-07-14T19:07:22.770 INFO:teuthology.orchestra.run.smithi088.stderr:user 0m1.534s 2023-07-14T19:07:22.770 INFO:teuthology.orchestra.run.smithi088.stderr:sys 0m0.078s 2023-07-14T19:07:22.772 INFO:teuthology.orchestra.run.smithi078.stderr: 90.9% -- replaced with /var/log/ceph/e2ca1a0e-2277-11ee-9b33-001a4aab830c/ceph-mon.a.log.gz 2023-07-14T19:07:22.772 INFO:teuthology.orchestra.run.smithi078.stderr: 2023-07-14T19:07:22.772 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m0.428s 2023-07-14T19:07:22.772 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m1.763s 2023-07-14T19:07:22.773 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.070s 2023-07-14T19:07:22.773 INFO:tasks.cephadm:Archiving logs... 2023-07-14T19:07:22.773 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi078/log 2023-07-14T19:07:22.775 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:07:23.085 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi088/log 2023-07-14T19:07:23.086 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:07:23.291 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi102/log 2023-07-14T19:07:23.292 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2023-07-14T19:07:23.489 INFO:tasks.cephadm:Removing cluster... 2023-07-14T19:07:23.490 DEBUG:teuthology.orchestra.run.smithi078:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c --force 2023-07-14T19:07:23.773 INFO:teuthology.orchestra.run.smithi078.stdout:Deleting cluster with fsid: e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T19:07:24.221 DEBUG:teuthology.orchestra.run.smithi088:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c --force 2023-07-14T19:07:24.521 INFO:teuthology.orchestra.run.smithi088.stdout:Deleting cluster with fsid: e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T19:07:24.976 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2ca1a0e-2277-11ee-9b33-001a4aab830c --force 2023-07-14T19:07:25.277 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: e2ca1a0e-2277-11ee-9b33-001a4aab830c 2023-07-14T19:07:25.734 INFO:tasks.cephadm:Removing cephadm ... 2023-07-14T19:07:25.735 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:25.756 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:25.778 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2023-07-14T19:07:25.796 INFO:tasks.cephadm:Teardown complete 2023-07-14T19:07:25.797 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-07-14T19:07:25.807 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-07-14T19:07:25.808 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:07:25.810 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:07:25.820 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-07-14T19:07:40.961 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-07-14T19:07:40.971 INFO:teuthology.task.clock:Checking final clock skew... 2023-07-14T19:07:40.972 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:07:40.975 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:07:40.978 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-07-14T19:07:40.992 INFO:teuthology.orchestra.run.smithi078.stderr:bash: ntpq: command not found 2023-07-14T19:07:40.994 INFO:teuthology.orchestra.run.smithi088.stderr:bash: ntpq: command not found 2023-07-14T19:07:40.996 INFO:teuthology.orchestra.run.smithi078.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:07:40.997 INFO:teuthology.orchestra.run.smithi078.stdout:=============================================================================== 2023-07-14T19:07:40.997 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +1106us[+1070us] +/- 75ms 2023-07-14T19:07:40.997 INFO:teuthology.orchestra.run.smithi078.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -5038us[-5074us] +/- 69ms 2023-07-14T19:07:40.997 INFO:teuthology.orchestra.run.smithi078.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 7 +4572us[+4572us] +/- 64ms 2023-07-14T19:07:40.998 INFO:teuthology.orchestra.run.smithi078.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:07:40.998 INFO:teuthology.orchestra.run.smithi102.stderr:bash: ntpq: command not found 2023-07-14T19:07:40.999 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:07:41.000 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2023-07-14T19:07:41.000 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 8 +1149us[+1095us] +/- 75ms 2023-07-14T19:07:41.000 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -5012us[-5066us] +/- 69ms 2023-07-14T19:07:41.000 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 8 +4602us[+4548us] +/- 64ms 2023-07-14T19:07:41.001 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:07:41.002 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-07-14T19:07:41.002 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2023-07-14T19:07:41.002 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 7 +1003us[+1003us] +/- 75ms 2023-07-14T19:07:41.002 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 -5150us[-5204us] +/- 69ms 2023-07-14T19:07:41.003 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 +4476us[+4422us] +/- 64ms 2023-07-14T19:07:41.003 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-07-14T19:07:41.004 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-07-14T19:07:41.018 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-07-14T19:07:41.018 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-07-14T19:07:41.029 DEBUG:teuthology.orchestra.run.smithi078:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:07:41.143 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:07:41.244 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-07-14T19:07:41.348 DEBUG:teuthology.orchestra.run.smithi078:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:07:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:07:41.383 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:07:41.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:07:41.418 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-07-14T19:07:41.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:07:41.455 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-07-14T19:07:41.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-07-14T19:07:41.538 INFO:teuthology.task.internal:Duration was 1866.044102 seconds 2023-07-14T19:07:41.539 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-07-14T19:07:41.550 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-07-14T19:07:41.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:07:41.553 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:07:41.556 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-07-14T19:07:41.600 INFO:teuthology.orchestra.run.smithi078.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:07:41.602 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:07:41.608 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-07-14T19:07:41.973 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-07-14T19:07:41.974 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2023-07-14T19:07:41.974 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:07:41.997 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2023-07-14T19:07:41.997 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:07:42.019 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2023-07-14T19:07:42.020 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-07-14T19:07:42.046 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-07-14T19:07:42.047 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:07:42.049 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:07:42.061 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-07-14T19:07:42.140 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-07-14T19:07:42.140 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:07:42.143 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:07:42.145 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-07-14T19:07:42.267 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-07-14T19:07:42.282 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-07-14T19:07:42.282 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:07:42.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:07:42.336 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-07-14T19:07:42.368 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-07-14T19:07:42.383 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:07:42.386 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:07:42.389 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:07:42.410 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2023-07-14T19:07:42.412 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2023-07-14T19:07:42.438 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2023-07-14T19:07:42.462 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:07:42.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:07:42.481 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:07:42.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:07:42.497 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2023-07-14T19:07:42.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-07-14T19:07:42.519 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-07-14T19:07:42.531 INFO:teuthology.task.internal:Transferring archived files... 2023-07-14T19:07:42.531 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi078 2023-07-14T19:07:42.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:07:42.570 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi088 2023-07-14T19:07:42.570 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:07:42.606 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038/remote/smithi102 2023-07-14T19:07:42.607 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-07-14T19:07:42.646 INFO:teuthology.task.internal:Removing archive directory... 2023-07-14T19:07:42.647 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:07:42.650 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:07:42.652 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2023-07-14T19:07:42.699 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-07-14T19:07:42.715 INFO:teuthology.task.internal:Not uploading archives. 2023-07-14T19:07:42.716 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-07-14T19:07:42.733 INFO:teuthology.task.internal:Tidying up after the test... 2023-07-14T19:07:42.734 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:07:42.737 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:07:42.739 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-07-14T19:07:42.751 INFO:teuthology.orchestra.run.smithi078.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:07 /home/ubuntu/cephtest 2023-07-14T19:07:42.753 INFO:teuthology.orchestra.run.smithi088.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:07 /home/ubuntu/cephtest 2023-07-14T19:07:42.757 INFO:teuthology.orchestra.run.smithi102.stdout: 262307 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 14 19:07 /home/ubuntu/cephtest 2023-07-14T19:07:42.759 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-07-14T19:07:42.775 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-07-14T19:07:42.807 INFO:teuthology.nuke:Checking targets against current locks 2023-07-14T19:07:47.844 DEBUG:teuthology.nuke:shortname: smithi078 2023-07-14T19:07:47.845 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:07:47.871 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:27:54.604030', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:07:47.878 DEBUG:teuthology.nuke:shortname: smithi088 2023-07-14T19:07:47.878 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:07:47.902 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:27:54.602748', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-07-14T19:07:47.910 DEBUG:teuthology.nuke:shortname: smithi102 2023-07-14T19:07:47.911 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-07-14T19:07:47.951 INFO:teuthology.orchestra.console:Power off smithi078 2023-07-14T19:07:47.953 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:07:47.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-07-14_16:44:47-fs:workload-wip-pdonnell-testing-20230714.134335-distro-default-smithi/7338038', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-07-14 18:27:54.601276', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='} 2023-07-14T19:07:47.982 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:07:48.007 INFO:teuthology.orchestra.console:Power off smithi088 2023-07-14T19:07:48.009 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:07:48.036 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:07:48.056 INFO:teuthology.orchestra.console:Power off smithi102 2023-07-14T19:07:48.057 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-07-14T19:07:48.084 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:07:52.026 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:07:52.070 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:07:52.111 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:07:57.140 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:07:57.183 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:07:57.223 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:08:02.257 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi078.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:08:02.297 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:08:02.335 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:08:02.470 INFO:teuthology.orchestra.console:Power off for smithi078 completed 2023-07-14T19:08:07.408 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:08:07.447 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:08:07.660 INFO:teuthology.orchestra.console:Power off for smithi102 completed 2023-07-14T19:08:12.524 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:08:17.636 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi088.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-07-14T19:08:17.850 INFO:teuthology.orchestra.console:Power off for smithi088 completed 2023-07-14T19:08:17.956 INFO:teuthology.run:Summary data: description: fs:workload/{0-rhel_8 begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/fuse objectstore-ec/bluestore-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/distributed export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/no 3-snaps/no 4-flush/no 5-workunit/suites/blogbench}} duration: 1866.0441024303436 failure_reason: 'Command failed on smithi078 with status 1: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph fs subvolumegroup pin cephfs _nogroup distributed 1''"' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=12a9a4a36cf644a8a96b101643743fc0 status: fail success: false 2023-07-14T19:08:17.956 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-07-14T19:08:21.504 INFO:teuthology.run:FAIL